punBB to MyghtyBoard importer

14 July 2008 Comments

It is possible to make quick and flexible migrations from PHP/Other sources to Django. We can use Django ORM as a data dump "format". If we do so we will get a dump that is database type indepented, and we wont have encoding problems (for non latin1 data). This article is based on older Diamanda release and the models aren't up to date.

I'll show here a punBB -> MygthyBoard migration script. First let's look at MyghtyBoard Categories model:

# Forum Categories
class Category(models.Model):
	name = models.CharField(maxlength=255, verbose_name=_("Category Name")) # name of the category
	order = models.PositiveSmallIntegerField(default=0, verbose_name=_("Order")) # order of categories on the forum-categories list
	class Meta:
		verbose_name = _("Category")
		verbose_name_plural = _("Categories")
		db_table = 'rk_category' + str(settings.SITE_ID)
	class Admin:
		list_display = ('name','order')
	def __str__(self):
		return self.name

We have to make a script, that will inserf few categories using Django ORM. Script like this one:

# -*- coding: utf-8 -*-
from os import environ
environ['DJANGO_SETTINGS_MODULE'] = 'settings'

from settings import *
from myghtyboard.models import *

p = Category(name='''Fora towarzyskie''', order='''90''')

p.save()



p = Category(name='''Rodzina BSD''', order='''60''')

p.save()



p = Category(name='''GNU/Linux''', order='''10''')

p.save()



p = Category(name='''jakilinux.org''', order='''120''')

p.save()

Place it in the Django project folder and execute. This scripts have 3 parts - core modules import, models import and last - data. Now we have to use PHP or Python with mysql-python (for MySQL) to make such script. It has to get all the categories from punBB and "show" them. PunBB categories table has 3 columns: id, name and disp_position (order), so all data is there. We need to print in a loop this code:

p = Category(name='''NAME''', order='''ORDER''')
p.save()

I used PHP and punBB API to do this:

<?php
<?PHP
define('PUN_ROOT', './'); // ścieżka do skryptu forum
require PUN_ROOT.'include/common.php';
ob_start();
echo '# -*- coding: utf-8 -*-
from os import environ
environ['DJANGO_SETTINGS_MODULE'] = 'settings'
from settings import *
';

IF($result = $db->query("SELECT * FROM ".$db->prefix."categories"))
	{
	echo "

".'from myghtyboard.models import *',"
".'Category.objects.all().delete()'."
";
	while ($row = $db->fetch_assoc($result))
		{
echo 'mc = Category(name='''.$row['name'].''', order='''.$row['disp_position'].''')
mc.save()'."
";
		}
	}

$wynik = ob_get_contents();
ob_end_clean();
file_put_contents('install.py', $wynik);
echo '<h1>install.py should be ready</h1>';

Place it in punBB folder and open in a browser. It will create a "install.py" file with all the categories ready to insert into MyghtyBoard.

Next one are forums. In this model we have ForeignKey:

# Forums
class Forum(models.Model):
	category = models.ForeignKey(Category, verbose_name=_("Forum Category")) # Forum category
	name = models.CharField(maxlength=255, verbose_name=_("Forum Name")) # name of the forum
	description = models.CharField(maxlength=255, verbose_name=_("Forum Description")) # desc of the forum
	topics = models.PositiveIntegerField(default='0', blank=True, verbose_name=_("Topics")) # number of topics
	posts = models.PositiveIntegerField(default='0', blank=True, verbose_name=_("Posts")) # number of posts
	lastpost = models.CharField(maxlength=255, verbose_name=_("Last Post"), blank=True, default='', null=True) # last poster info etc.
	order = models.PositiveSmallIntegerField(default=0) # order of forums on the category list

punBB table contains ID od the category entry. We have to use an object: Category.objects.get(id=ID_NUMBER). Also if our categories in punBB aren't numbered from 1 without any holes then we would also have to add original ID to the category dump. Here is the code:

<?php
IF($result = $db->query("SELECT * FROM ".$db->prefix."forums"))
	{
	echo "

".'Forum.objects.all().delete()'."
";

	while ($row = $db->fetch_assoc($result))

		{

echo 'mf = Forum(id = '''.$row['id'].''', category = Category.objects.get(id='.$row['id'].'), name = '''.$row['name'].''', description ='''.$row['desc'].''', order='''.$row['disp_position'].''', posts='''.$row['num_posts'].''', topics = '''.$row['num_topics'].''')

mf.save()'."

";

		}

	}

We used the category object. We also use the original ID for the forum entry.

The last problem is user import. Django saves user passwords like this

"sha1$SALT$HASH"

Where sha1> is the has function, SALT - random string added to the password to make the HASH. punBB uses by default sha1 so there is no problems with user import. We just have to use empty salt:

"sha1$$HASZ"

The migration code would look like this:

<?php
IF($result = $db->query("SELECT * FROM ".$db->prefix."users WHERE id > 2 AND num_posts > 10 ORDER BY last_post DESC"))

	{

	echo 'User.objects.filter(id__gt=1).delete()'."

";

	while ($row = $db->fetch_assoc($result))

		{

		echo 'us = User(username="'.$row['username'].'", first_name="'.$row['username'].'", last_name="'.$db->escape($row['username']).'", email="'.$row['email'].'", password="sha1$$'.$row['password'].'", is_staff="0", is_active="1", is_superuser="0", last_login="2006-08-29 20:00:34", date_joined="2006-08-29 20:00:34")'."

";

		echo 'us.save()'."

";

		}

	}

Results

Full code:

<?php
<?PHP

define('PUN_ROOT', './'); // ścieżka do skryptu forum

require PUN_ROOT.'include/common.php';

ob_start();

echo '# -*- coding: utf-8 -*-

from os import environ

environ['DJANGO_SETTINGS_MODULE'] = 'settings'

from django.contrib.auth.models import User, Group

from settings import *

';

IF($result = $db->query("SELECT * FROM ".$db->prefix."users WHERE id > 2 AND num_posts > 10 ORDER BY last_post DESC"))

	{

	echo 'User.objects.filter(id__gt=1).delete()'."

";

	while ($row = $db->fetch_assoc($result))

		{

		echo 'us = User(username="'.$row['username'].'", first_name="'.$row['username'].'", last_name="'.$db->escape($row['username']).'", email="'.$row['email'].'", password="sha1$$'.$row['password'].'", is_staff="0", is_active="1", is_superuser="0", last_login="2006-08-29 20:00:34", date_joined="2006-08-29 20:00:34")'."

";

		echo 'us.save()'."

";

		}

	}





IF($result = $db->query("SELECT * FROM ".$db->prefix."categories"))

	{

	echo "



".'from myghtyboard.models import *',"

".'Category.objects.all().delete()'."

";

	while ($row = $db->fetch_assoc($result))

		{

echo 'mc = Category(name='''.$row['name'].''', order='''.$row['disp_position'].''')

mc.save()'."

";

		}

	}



IF($result = $db->query("SELECT * FROM ".$db->prefix."forums"))

	{

	echo "



".'Forum.objects.all().delete()'."

";

	while ($row = $db->fetch_assoc($result))

		{

echo 'mf = Forum(id = '''.$row['id'].''', category = Category.objects.get(id='.$row['id'].'), name = '''.$row['name'].''', description ='''.$row['desc'].''', order='''.$row['disp_position'].''', posts='''.$row['num_posts'].''', topics = '''.$row['num_topics'].''')

mf.save()'."

";

		}

	}

	

IF($result = $db->query("SELECT * FROM ".$db->prefix."topics"))

	{

	echo "



".'Topic.objects.all().delete()'."

";

	while ($row = $db->fetch_assoc($result))

		{

echo 'mt = Topic(id = '''.$row['id'].''', topic_forum = Forum.objects.get(id='.$row['id'].'), topic_name = '''.$row['subject'].''', topic_author =  '''.$row['poster'].''', topic_posts =  '''.$row['num_replies'].''', topic_modification_date =  '''.date("Y-m-d H:i:s", $res['posted']).''')

mt.save()'."

";

		}

	}



IF($result = $db->query("SELECT * FROM ".$db->prefix."posts"))

	{

	echo "



".'Post.objects.all().delete()'."

";

	while ($row = $db->fetch_assoc($result))

		{

		$row['message'] = nl2br($row['message']);

		$row['message'] = str_replace("

", '', $row['message']);

		$row['message'] = $db->escape(str_replace("



", '', $row['message']));

echo 'mp = Post(post_topic = Topic.objects.get(id='.$row['topic_id'].'), post_text = '''.$row['message'].''', post_author = '''.$row['poster'].''', post_date = '''.date("Y-m-d H:i:s", $res['posted']).''', post_ip = '1.2.3.4')

mp.save()'."

";

		}

	}

$wynik = ob_get_contents();

ob_end_clean();

file_put_contents('install_1.py', $wynik);

echo '<h1>install_1.py should be ready</h1>';

?>

example punBB

migrated data in MyghtyBoard

RkBlog

Diamanda Applications Set, 14 July 2008