BuddyPress

OK, here are the goods: http://teleogistic.net/2010/07/import-from-ning-now-imports-ning-content-into-buddypress/

Special thanks to Leo, Caspar and Lorraine for generously loaning me their Ning network data for testing! No guarantees that it will be 100% perfect, but without their help I never could have gotten as close as I have.

Let me know how it works out for you!

Boone,

I tried to import the data on 2 different servers, and 1 I am getting this error :

http://dl.dropbox.com/u/253979/error.png

and on the other, the script ran it imported 3 members and stopped, if I refresh or run the importer again I will get this and nothing else:
Importing users – this may take a while.
If you see a Refresh message at the bottom of the page you’ll need to refresh the page in order to continue importing.

Thanks,

Leo

Leo – I had a funny loop in there that might have been causing a problem with member imports. Please update the plugin to v2.0.1 and run it again (and thanks for the bug report).

boone – it seems that everytime I run the script on a refresh, its going through 1-2 users, and I have to refresh again – version 2.0.1

Leo – are you on shared hosting? Do you have any idea what your PHP memory limit is?

Boone I will take a further look at it tomorrow, but I raised the php memory limit in one of them to the max, and the problem persisted. But I will keep you informed.

Thanks

Hi Leo – Thanks for having a look. Using your data on my local machine (BP 1.2.5.1 and WP 3.0), Import from Ning 2.0.1 imports all of your users with just a single refresh. But it does make my computer fans kick on!

If you continue to have problems, you might try dividing up the members file and running the member import in multiple stages. Just make sure that you keep the JSON well formatted. That means making sure that you divide the file only between member entries. Members are divided by the following code: },{. Be sure that you keep the same punctuation at the beginning and end of the document as well. Because of the way that the document is loaded into memory, I’m guessing that a member file that is half as large will need *less than half the memory* to run properly. So it’s worth a try.

A little update. I just did some testing on my local machine where I reduced WP’s memory_limit to 32MB (typical for shared hosting). Surprisingly, it didn’t slow things down that much – 1300 users were imported in five refreshes rather than one or two.

Then I opened a top window to monitor CPU processes. It turns out that, during the creation of members, CPU spikes (on my brand new, souped up Macbook Pro) to well over 50%. Most of that comes from mysqld, which is not surprising given that the import process requires so many updates to the database, which are cpu-expensive. So the bottleneck could be the CPU limit given to you by your host.

I don’t know of a way around this, but I’ll think about it.

Boone,

I will take a look to see what happens if I split the json file, also I will try to give a run of the importer on my dedicated server, but I was thinking if there is a possibility for you to import to run the importer in a way that does 5 members at a time, until completed.Also I remember the first time I ran the test plugin it importer 500 members no hiccups.

I will take look at it and let you know.

Thanks again.

Hey Leo – I’m comparing the two versions, and the only difference I can see in the member import process that might be causing a hangup has to do with avatar importing. I made a change for the release version to ensure that the avatar cropping process actually went through, because in some of my tests the avatar was just getting skipped. But it’s possible that the check is actually causing your problem

Can you test the following? In bp-functions.php of 2.0.1, replace lines 236-240 with the following line:
bp_core_avatar_handle_crop( $args );

I’ll do some more experimentation on my end too. Thanks for your help and patience – it’s hard to test these things in all possible environments.

Boone Good News! After making the changes on the bp-functions.php all the users were created, in 2 refreshes even though I got the message about line 210 of the bp-functions.php,

X-Profile:
after it ran the x-profile fields I got the following message

warning: /home/content/l/e/o/leoafarias/html/wp-content/plugins/buddypress/bp-xprofile.php on line 707

Warning: Missing argument 3 for xprofile_set_field_data(), called iHTTP/1.1 200 OK Date: Sat, 24 Jul 2010 15:22:59 GMT Server: Apache Expires: Wed, 11 Jan 1984 05:00:00 GMT Last-Modified: Sat, 24 Jul 2010 15:22:59 GMT Cache-Control: no-cache, must-revalidate, max-age=0 Pragma: no-cache Keep-Alive: timeout=15, max=97 Connection: Keep-Alive Transfer-Encoding: chunked Content-Type: text/html; charset=iso-8859-1

after another refresh, it seems like i kept getting the warning but the profile fields were imported.

Groups:
The first time I ran I got a fatal error regarding the bp-forums inside plugins/Buddypress, then I restarted the importing process again, and the second time it went through after 5 Refreshes

Discussions:
I got this message:
No forum id – skipping
Refresh to import more discussions

I refreshed a couple of times, and the same message kept showing up.

This is pretty much where I am at, I was looking at the import, and I have all the member avatars, the profile fields so far seemed to be imported fine also, but I do not have the GROUP’s Avatar. I am not sure if the importer was supposed to that or not.

Thanks again for all your work

Boone also it seems like the friendships were not imported. The users friend count show 0.

Cool! Glad we’re making progress.

The plugin is not currently set up to import group avatars. There were a few complications that kept me from making it happen. I’ll add it soon (and you should be able to run it again without starting from scratch).

The plugin also doesn’t handle friendships. Honestly, I didn’t know that there were friendships in Ning! I will have to look more closely at the json dumps in order to see how it’s encoded. If it’s there, then I must have missed it.

As for the discussions: That error suggests that something has gone wrong with the way that groups and discussion categories were imported. When those imports happen, I create a piece of data that matches the unique Ning group ID (usually a number, but in the case of discussion categories it’s the category name) with the BuddyPress group ID, then I save that array to the database. It’s possible that this indexing array is totally missing, or that it’s incomplete. The other possibility is that the group ID has been stored correctly, but the corresponding forum wasn’t created correctly It’s a bit hard to tell without seeing your database.

In any case, I should be able to add a double checking mechanism that corrects that group ID index when it hits a discussion with no forum ID attached, and also ensures that there is a forum created for that group. I might get a chance to code that tomorrow. I’ll post an update when I do. In the meantime, find the line that says { echo "No forum id - skipping"; } in bp-functions.php and change it to { echo "No forum id - skipping"; continue; } (which is what it should be anyway) and try the import again. It’ll skip that particular topic but move on to the next one.

Alright Boone,

I was able to get a full import of the ning network:
Final Stats
1317 members
339 updates
163 groups
161 forums
129 topics

After made the changes that you suggested Boone, the plugin ran amazingly smooth all the way through!
Amazing.

That’s fantastic news, Leo!

I’m looking at a bunch of Ning export files right now and I can’t find friend data *anywhere*. Weird! Going to work on the rest of the fixes today or tomorrow. Thanks again for all your help!