I also grew up in Sweden (I live there) so I know full well about the english. However I am concerned because English holds many similarities to swedish and thus it is easy for swedes to learnt it, Japanese does not.
I watched anime and (tried) to read some simple japanese but if you don't have someone explain it to you learning anything becomes extremely difficult.
Ofcourse it's not my intention to bring you down or anything and you might actually succeed. . . I'm just saying that if you have the option of formal learning, you will better understand what you are listening to.
Thanks~