Reply
May 17 2011
By: PipenAin Hekseville Citizen 299 posts
Offline

PSN OUTAGE:Fully analyzed/explained

4 replies 3864 views Edited May 17, 2011

Most are aware of the circumstances, but to those that are confused, I will attempt to summarize the events that lead to the removal of online services. I will also analyze popular theories about the causes of the outage, as well as why its taking so long to restore... To those that are allergic to "walls of text", you might want to skip this post entirely. The last paragraph is where I make this related to Home, and to all else looking for an explanation as to why they lost access to PSN for 24 days, the rest of this post is relevant as well. The historical links are not meant to be discussed, they are included only as reference, but feel free to post your own credible links in the comments.

We know more now than when I originally posted this on "playstationhometoday.com", and since ive got allot of comments about the helpful nature of gathering links, ill include a couple more. I'll start with a few possible numbered reasons why it took so long restore PSN. Afterwards, I will continue with a lengthy history of the cyber war Sony has been fighting.

1. Sony moved its AT&T Data Center to a new location, most likely an entirely new country. Details can be found here:

http://blogs.vanderbilt.edu/~terrence.brown/?p=7

One might assume this to be an indication of an internal employee being associated with the hack, but in actuality Sony already had the new location purchased and planned months prior to the event. Since they needed to rebuild the security anyways, the outage seemed to be the most logical time for them to transfer headquarters.. As you can imagine this must be a very time consuming process all by itself, even without the added task of preventing all future breaches.

2. As the blog states, there was "an external intrusion" into the private servers of PSN. Later, they reference a group of hackers known as "ANONYMOUS" with a history of hiting other major coporations with "denial of service" attacks, with many members already being busted by the FBI earlier in the year for other cyber attacks unrelated to Sony. They are most likely still occupying Sony servers with countless log-in requests as well as possibly leaving dozens if not hundreds of viruses to clean up. The discovery of the file "Anonymous, We Are Legion" is an indication that they (or someone) had the motivation to plant data onto their servers, in an attempt to occupy their employees trying to discover and remove the files.

References to previous arrests of 5-6 Anonymous members as well as 40 FBI warrants for anonymous arrests can be found here:

http://v3.co.uk/v3-uk/news/2030633/fbi-seeking-anonymous-uk-arrests

http://baltimorenonviolencecenter.blogspot.com/2011/04/new-arrest-over-anonymous-pro-wikileaks.html

3. A decision was most likely made by a group of Sony executives to further delay the restoration of PSN regardless of it actually functioning already, because they added the additional task of developing and testing new MAJOR changes/improvements to PSN. Without having to use resources monitoring millions of players they would be able to experiment with a few things like cross game chat and voice messages. They probably argued that one or two extra weeks would be eventually ignored if popular requests were finally implemented onto the network during this down period. For public relation reasons they also decided not to confirm this as their current project because impatient kids will scream and cry to simply restore PSN at the cost of new features, but a sony rep did reference this as one of their side projects. The sony rep admits that the features might not be "right away", but that customers will be pleasantly surprized in the days following the restoration of the services.

Details on partial confirmation of cross game chat/voice messages can be found here:

http://www.vgsynergy.net/2011/05/07/you-will-be-surprised-and-pleased-with-the-new-psn/

4. While its true they were "totally rebuilding" PSN, much of the old Network data had to be used again. Obviously they couldnt just totally rebuild from scratch, because they had the neccessary hurdle of also including their customers history. The fact they had to use our old data might have made it slightly more time consuming for them. This includes our purchase history, ban history, friends list, trophies, cloud saves, etc. If any of the data got corrupted along the way or if there were viruses compromising the data, that all had to be resolved multiplied by millions of accounts.

Ok, now for a bit of history leading to the hackers involvement in PSN. Sorry for the long post but id rather be detailed than to be brief...

For a list of terrorist style videos created by ANONYMOUS, check out the link below. They have done several other videos but they are still in the process of uploading them onto this new web hosting site chosen because Youtube may soon be another target of theirs. (the style of the threat videos are very similiar to the "SAW" movies where an unknown identity passes vigilante judgement on another, promissing torture/death):

http://www.dailymotion.com:80/anonyops#videold=xiauce

Its possible that this link may eventually be temporarily disabled because two Anonymous sites were hacked recently by one of the former moderators of the group known as RYAN. He argues that the group deserved to be targeted because of the lack of true leadership and organization, and for denying participation in the Sony scandal despite previously rallying their members and the public to take action against them.

More information on the Anonymous splinter group hacking their own sites can be found here:

http://www.geek.com/articles/news/anonymous-irc-servers-hacked-by-a-splinter-group-20110510/

Anonymous has a history of attacking any corporation known to be anti "Wiki-leaks"  including major firms like Bank of America and Amazon.com ..Wiki-leaks are posted by another group leaking government/CIA stolen secrets, and Anonymous claims that denying others the right to leak information violates their "freedom of speech". Even going as far as attacking the military for jailing the supposed original theif of government secrets. They also targeted the group of Scientology for violating human rights, also a Baptist website for attempting to brain wash society into hating homosexuals, they targeted Australian/German government for restricting access to certain websites to their public, "youtube.com" for filtering videos, and have targeted a handful of other random targets like Animal abusers, child pornographers, the RIAA, "paypal" and more... They have even most recently targeted Koch in canada as well as the government of IRAN.

Theres countless sources detailing their past targets, so ill update this section with a few of those sources. The first link summarizes a decent amount of their history as well as explaining the connection with the website "*****.com" where this group evolved from:

http://digitaltrends.com/computing/*****-based-group-anonymous-targets-paypal-to-support-wikileaks

http://Techgeek.com.au/2011/02/21/anonymous-targets-westboro-baptist-church-church-says-bring-it/

http://ethicalactionalert.com/2011/02/anonymous-targets-brothers-koch.html

http://cbc.ca/news/technology/story/2011/03/10/anonymous-manning-quantico-ddos.html

http://inquistr.com/36559/anonymous-targets-australian-government-over-internet-censorship

http://readwriteweb.com/archives/anonymous-targets-iran.php

And most recently on April 5th they began a vigilante crusade against Sony for supposedly denying consumer rights to mod their ps3's (comparing it to adding rims to a car you purchased) and also in retaliation for Sony geting a court order to retrieve all IP's and Home addresses of anyone visiting hack related websites or GeoHotz "jailbreak" youtube page. Sony justified needing those addresses as a way to further convict GeoHot, not in an attempt to bust anyone else discovered viewing youtube or using Geo's site. This is because there are extra laws in California related to DATA CRIMES, so they wanted to confirm how many from California his exploit spread to.

Confirmation that Sony was granted the rights to collect personal information of anyone visiting GeoHotz youtube page as well as other hacker sites can be found here:

http://forums.whyweprotest.net/threads/sony-gets-ip-addresses-of-anyone-who-visited-ps3-hacker%E2%80...

While its true that the "hacktivist" group called Anonymous targeted Sony with a terrorist style threat on April 5th/9th, it should be known that there was also a group of sony fanboys/fangirls that experimented with new uses of GeoHotz JAILBREAK during this same scandal period. Between April 1st to April 20th the JAILBREAKERS found a way to create a "Developer Account". As a Dev account, they were able to bypass the normal verification/security processes that normal accounts have, which lead to their ability to hack FREE games from the PS STORE using fake default credit card details.

This activity was confirmed by a european sony rep in this link:

http://www.psu.com/PSN-hackers-attacked-Admin-dev-accounts,-services-returning-i-a-day-or-two,-SCEE-...

A more detailed description of the impact can be read here:

http://www.escapistmagazine.com/news/view/109545-speculation-about-psn-outage-turns-to-custom-firmwa...

Despite the unrelated intrusion into Developer privileges, Anonymous is still partially to blame for the overall decision to shut down servers to rebuild it. In the least they were involved by occupying Sony from the 5th to the 20th with a classic DENIAL OF SERVICE attack using the Orbit Ion Cannon program (designed to overload a system with countless log-ins/requests). Most were unaffected by this attack, while others recieved constant sign-in errors or were booted from the server randomly. They even temporarily targeted Sony's websites Sony.com PlaystatonLifeStyle.net playstation.com and the SOE site. An investigation into the extent of the impact while on those sites was done later by Sony, the FBI, and forensic teams which lead to exagerated claims of credit card data being stolen from SOE/PSN because they had access to the main server containing that info. Its true that someone posted non vital "personal information" on hacker forums/sites but it was quickly removed, and it only contained outdated useless information like date of birth, name, and address. They argue that the potential exposure of the basic details was partially as freedom of speech in the form of a "leak of the truth", as well as to occupy Sony in their attempts to clean up their "denial of service attack".Some panic over the idea of a shared address but do you panic because your neighbors know where you are, or that you can be found on GoogleMaps? Thats why we have police as a deterrent to criminals trying to simply kick every door down in the world.

Details related to someone leaking SOE website data can be found in this link here, and a reference to them considering a BOUNTY TO CATCH THE HACKERS is also mentioned:

http://www.gossipgamers.com/sony-removes-exposed-personal-data-delays-psn-restart/

It was during this initial attack by Anon where they most likely left the file Sony referenced in the blog with the quote "Anonymous, We are Legion". While Sony's tech team worked daily to resolve that issue, the jailbreakers had more opportunity to breach the system undetected. And as previously stated, the fact that they actually left a file behind is an indication that they could have left much more like viruses. Anonymous even state in their videos that the Ion Cannon Denial of Service attack isnt the only weapon in their arsenal, but their goal remains unchanged. They have never been known to seek financial profits from their hacks, instead their goal is to torture a corporation untill a list of demands are met under the supposed umbrella of either protecting free speech or attacking those that deny it.

At this point id like to stress a major point. It should be recognized that Sony never guaranteed that our personal information was taken during this breach, only that it was a possibility. After tracking the origins of the attack, they saw the possibility that the group could have extended the dev account exploit to reach the main servers containing our personal data (name, address, date of birth, hashed password, and credit card billing info without the PIN code) Also anonymous had access to sites with similar info. However theres no evidence that our info was actually used or even seen, despite unreliable claims of a list of our info being sold or passed around on hacker forums...

And without a social security number or credit card PIN code, theres very little they can do with the information anyway. And in the unlikely scenario that anything was used, Sony is giving us "identity theft protection" soon from DEBIX (other regions to get similar programs too). Plus we agreed to a Terms of Service which frees them from any and ALL blame/lawsuits.

The originator of the "JAILBREAK" GeoHot, denies association with Anonymous as well as the events that lead to the shut down of the server. The supposed original intent of the jailbreak was to play moded games offline, and to run custom aps on the ps3, and to run an alternate Operating System as a different choice to the standard XMB and web browser. Geo argued that hacks can coexist in a legit community if a server is provided to filter them out of the main population like what is commonly done for PC game exploiters. Since its too time consuming and nearly impossible to constantly patch new methods, its far easier and cheaper to simply allow exploits to be done in a private community at the risk of permanent ban if done in the main population. But on the PS3 you dont get a choice, its either play legit online or mod offline at risk of copyright laws. GeoHotz was already famous for cracking the Iphone/Ipad "jailbreak" method, and focused on Sony till he discovered a way last year despite a clear indication of the rules prohibiting us.

I dont agree with GeoHotz decision to circumvent the rules, he should have brought to court or petitioned Sony executives to defend his point of view rather than bypassing the current firmware, as well as posting to all on youtube how to do it. The simple fact that he chose fame over safety showed a lack of common sense. Also, he did develop the method to download digital versions of blu-ray games which he defended as a form of data backup in case a disc breaks, which of course lead to piracy of brand new games like KillZone3. Sony understandably decided to sue him for MILLIONS and even threatened with jailtime earlier this year. Around March they settled the case outside of court, giving him a much smaller dollar penalty, no jail time, and had him sign a injunction contract promissing to never use a PS3 again. However I must say that even though Sony was justified, they also took a BIG risk in targeting a community of focused hackers without having a contingency plan if they retaliated.

And it doesnt make much sense to me why they wouldnt pursue stiffer penalties versus GeoHotz since they already exposed the whole issue to the media anyways with the lawsuit, and because they would need to bust someone as a form of establishing precedent towards penalities of any future hackers. It leads me to believe the gathering of Home addresses and IP addresses of people viewing his sites was in fact not for the lawsuit, but instead the goal was to monitor those actively involved in hacking PSN in order to give their tech team data helpful in preventing future breaches.

The Developer Account exploit was not created by GeoHotz however, because he was in South America during its development and because it has no benefit to him now that he's permanently banned at risk of stiff legal penalties. GeoHot comments on the current situation a bit in this link:

http://www.escapistgmagazine.com/new/view/109650-geohot-sounds-off-on-sonys-psn-debacle

One final thought about the nature of the network prior to April 20th. There have been rumors that it was extremely easy to hack because it was runing on an outdated version of Apache Server with NO FIREWALLS. This is totally untrue. Evidence against those claims can be seen in the Tokyo Press Conference by Kaz Harai. After the asian style head bowing apology, they reference a diagram showing THREE FIREWALLS used as backup in case one or two get breached. But as we know, no system is perfect and anything can be analyzed and bypassed in time.

Evidence denying the outdated Apache Server claims can be seen at this link, where an investigator uses his history on PS3 internet browser to determine the details of what type of server it is:

http://www.bitmob.com/articles/detective-work-reveals-psn-servers-up-to-date

The only reason our browser on the ps3 sucks is because its missing current FLASH and is using a strange custom browser similiar to Internet Explorer.

*****************************************************************************

Anyways, I hope this was an interesting read for anyone confused about some of the rumors and about the origins/history of the hack groups. I tried to only include credible links and those with direct quotes rather than interpretations...  I'm looking forward to seeing the accumulated updates we've been missing in PS HOME!! Thankfully it became true that "FULL" services wont return untill as late as May 31st, but its great we got access to everything else except the PS STORE. As of the time of this post, Japan home isnt as lucky as our region and Europe because they are still waiting for even partial services. News from japan has revealed that their govenment is applying extra pressure on Sony to make sure that the network security is at the maximum possible levels, to make comparable to Banking or Government data protection..

Any day now, the playstation store will become available on the XMB as well as the HOME Store. I would be surprized if it returned as late as May31st because its a serious priority right now. Not just for the customers with itchy wallets, but mainly to help appease the game developers still waiting on daily PSN sales to roll in. And ideally HOME can incorporate its own "welcome back" program in the form of REWARDS or items priced at 0.00 in the Home Mall when it updates. Currently, the version of Home we have access to now is the same that was available prior to the psn outage, including message of day.

In the very least, I seriously expect at least a WELCOME BACK EVENT in the form of a virtual banner, or fireworks, or some other sort of visual display. Even a single free item would be greatly apreciated by the hardcore collector types as well as newbs. If not, this will be a huge missed opportunity. I do understand that many developers are concerned that they have already lost "expected daily income", but sometimes you have to spend money to make money when in business. Since most companies are already working on things, it would only require minimal effort by a developer to adapt a future item towards a WELCOME BACK EVENT, and would include them in the "customer patience apreciation" idea that Sony is already working on (possibly giving them the chance to be mentioned by some tech website summarizing the overall benefits of the return). The collectors will still buy things regardless how many free items become available in the future, so I sincerely hope they turn a deaf ear to any greedy accountants that might argue against the idea of anything being free at this time.

**EDITED** I recently read the blog post on the Customer Apreciation program, and am greatly pleased that at least 100 items are expected to be part of the package as well as reference to some sort of game and the mansion add-on space (most likely access to the addon wont be available unless you also own the Mansion 1st floor for 15$)...  KUDOS to the developers for puting customers before "potential profit losses". It remains to be seen what sort of items will be included. Most likely it will be mainly furniture items, but there since its 100+ items it sounds like there will at least be a few decent wardrobe items. Hopefully they arent all cheasy t-shirts in various colors reading "i survived the psn outage" or whatever, but even that is pretty cool!!

Also it should be understood that the details of the Customer Patience Apreciation program are still premature. Its entirely possible that they made include a handful of extra things as a surprize, since we're already prepared to recieve less. If not another game, even the inclusion of cheap friends list avatar pictures and themes would be very cool and easy to implement. Also it would be nice if they say we can get ALL AVAILABLE games on the list rather than picking just two because many people have alt accounts anyways so why not.

Get">http://www.us.playstation.com/psn/signup">Get your Portable ID!

Get your Portable ID!

Invisibility in pic below occured naturally during a random error within Home a couple years ago! Me and my friends shirts/pants appeared this way upon login.
Message 3 of 5 (3,864 Views)
Reply
0 Likes
Wastelander
Registered: 07/18/2009
Offline
682 posts
 

Re: PSN OUTAGE:Fully analyzed/explained

May 16, 2011

One element to add: apparently the hackers used fake credentials and established an Amazon EC2 cloud account and launched the theft/penetration attack from there, not a botnet of zombie pcs.

This race to the cloud is not without drawbacks, as far as I am aware, this is precedent for a commercial cloud service provider being the launching point for computer crime.

The "leaders" of Anonymous are discovering the drawbacks of a loosely affiliated, decentralized group: no solid control. So their denials of anything to do with the breech are bunk as they have no way of knowing who all the people are who identify themselves as part of Anonymous nor what they are doing.

Message 1 of 5 (3,864 Views)
Reply
0 Likes
Wastelander
Registered: 05/02/2008
Offline
748 posts
 

Re: PSN OUTAGE:Fully analyzed/explained

May 17, 2011

PipenAin wrote:

The simple fact that he chose fame over safety showed a lack of common sense. Also, he did develope the method to download digital versions of blu-ray games which he defended as a form of data backup in case a disc breaks, which of course lead to piracy of brand new games like KillZone3.


While I myself hate some of the restrictions on backing up media ..this is to include DLC like PAIN and Borderlands.  I would love to have a feature to CHOOSE WHAT CONTENT I WAN TO BACK UP.  There is nothing more frustrationg then spending an hour or more (which I dont have that problem @50 Mbps both ways) downloading a game then start it ONLY TO FIND that you now will be waiting most of your day for the update to download and install.  The issue; the DLC is NEVER updated to a current refresh.  Such as with Pain, you download v1.0 and I beleive we are upto 7.6 or something.  If we could backup 'select content', vs waiting for the entire backup process which takes more than forever on my 500 GB's; that would be AWESOME!  It would also serve a purpose for hard media as well.  But facts is facts; even though according to copyright laws we should be able to have one copy in case of accident of all our media ...its why Slysoft exists... many people abuse the software for not so great purposes.

PipenAin wrote:

One final thought about the nature of the network prior to April 20th. There have been rumors that it was extremely easy to hack because it was runing on an outdated version of Apache Server with NO FIREWALLS. This is totally untrue. Evidence against those claims can be seen in the Tokyo Press Conference by Kaz Harai. After the asian style head bowing apology, they reference a diagram showing THREE FIREWALLS used as backup in case one or two get breached. But as we know, no system is perfect and anything can be analyzed and bypassed in time.

Evidence denying the outdated Apache Server claims can be seen at this link, where an investigator uses his history on PS3 internet browser to determine the details of what type of server it is:

http://www.bitmob.com/articles/detective-work-reveals-psn-servers-up-to-date

The only reason our browser on the ps3 sucks is because its missing current FLASH and is using a strange custom browser similiar to Internet Explorer.


What was revealed here is that ALTHOUGH the outside (persay) servers had firewalls and updated software, several of the connecting servers (all seemingly in-house) did not and WERE running Apache Unix from 2008(latest update).  Whether you like it or not, the fact that these servers, mainly dealing with RETAIL CONTENT authorization and login process, were in some fashion connected to an 'outside' line via ANY fully updated main page server left certain ports exploitable thru packet passing.

I will give you four stars for the DEFINTE 'fully analysed' part but you didnt do much for explaining.  Sadly, trying to put helpful content like that here is generally frowned upon ...WTF?!?  Sony obviously was behind on their education yet we cant come here to support our products safety???

Read the tags of the following post:

http://community.us.playstation.com/thread/3422050

The exploit that is mentioned attacks in the exact fashion EVERYTHING happened ...yet was posted ONE WEEK BEFORE 4-20?  This was due to my ownb research and strangeness in my own account BEFORE the said 'intrusion' actually took place.  I just found out the thread was locked for MUCH less than I see happening ALL OVER THE BOARD but hmmm, maybe its says something, eh.

You couldnt help your greedy lil fingers...now its a G2 thang.
Message 2 of 5 (3,864 Views)
Reply
0 Likes
Hekseville Citizen
Registered: 09/06/2008
Offline
299 posts
 

Re: PSN OUTAGE:Fully analyzed/explained

May 17, 2011

Very good points HyJaxTD. And yes some of the explaination of the hack details was reduced so as not to encourage others to experiment and to prevent the post from being blocked. But its a bit harsh to say i didnt "explain", while at the same time you agree that I "analyzed" it. An analysis and explaination are almost synonyms. And afterall, I didnt name the post PSN HACKER ACTIVITY: Fully analyzed/explained.. 

Also, in my opinion I did attempt to explain both Sony's and hackers points of view, including GeoHotz original intent of downloadable Blu-Rays being a supposed non malicious attempt to have a copy of your own purchased material in case of damage (or potentially for speed issues as you mentioned). But its untrue that its totally allowed by copyright law, the list of restrictions on cd is soon to extend to dvd/blu-ray, and what the jailbreakers were doing was freely distribuiting it allowing anyone capable of the exploit to download a copy of games they never owned to begin with.

A brief description of the current situation towards copyright law changes can be seen here, mainly because of applications like Slysoft charging customers to gain the ability to bypass normal restrictions. Obviously they would eventually fight this because they put those restrictions there for a reason:

http://en.akihabaranews.com/76455/internet/jaca-considers-expanding-copyright-law-to-dvds-cant-learn...

Your description of the servers is accurate as well, but as mentioned it was mainly a flaw on Sony's side of the server which was never fully expected to be breached. They were aware of the potential, but to totally overhaul the system is an extremely time consuming task. Many customers became infuriated after only 1-2 days downtime, let alone the 24 days it took to eventually bring things up to higher standards.

Its true there were obvious holes in the system, but those are tiny holes in something as huge as "the wall of china". The fact that an isolated group of clever daily determined hackers found ways to creep into those holes doesnt mean that the system was entirely flawed garbage. Only a very small percentage of the overall community was aware of the hacks or brave enough to attempt them. But I did mention that it surprized me that they wouldnt have already updated their system after making the decision to attack GeoHot and all other hackers visiting his sites. Ideally, they could have shut down the servers for 3-6 hours a day untill it was further secured.

Invisibility in pic below occured naturally during a random error within Home a couple years ago! Me and my friends shirts/pants appeared this way upon login.
Message 4 of 5 (3,864 Views)
Reply
0 Likes
Highlighted
Hekseville Citizen
Registered: 09/06/2008
Offline
299 posts
 

Re: PSN OUTAGE:Fully analyzed/explained

May 17, 2011

Yes thats true radix, that was part of the jailbreak exploit allowing them to escalate privaledges and to bypass normal security. Theres allot of other details as to what they specifically did which i chose to leave out of the post because it only baits people into taking foolish risks.. 

And about the single Anonymous denial video. Its almost like they are denying an assasination after making SEVERAL videos to rally their troops into action. After their first 2 hate mongering videos, they would have to be morons to not expect their followers to pursue it.. In my opinion the denial was only an attempt to claim plausible deniability in case they are later caught by the FBI again.

Also, their denial was using clever word usage that didnt specifically deny that they hacked Sony. They said that they arent responsible for the PSN outage/shut down of the server.. Well DUHH!!! We all already know that Sony is responsible for the decision to remove services AFTER they were hacked or "externally intruded". The fact that they claim that they didnt shut down the server entirely is an irrevelant point, because they never confirm that they didnt initiate a denial of service attack or viruses / file insertions which eventually contributed to Sony's decision to shut it down entirely.

Anonymous has followed through with every other video threat they've made, so why would they stop. Then by coincidence the server gets shut down within a couple weeks of their initial videos on April 5th and April 9th. Fail. Innocent or not, you cant make threats and expect total innocence, especially when associated with peers that have a long history of hiting highly secure corporations/governments.

Invisibility in pic below occured naturally during a random error within Home a couple years ago! Me and my friends shirts/pants appeared this way upon login.
Message 5 of 5 (3,864 Views)
Reply
0 Likes