Study MongoDB administration

Today we are going to look into MongoDB administration. We will focus on a few areas, the backup, monitoring, configuration, import and export data.

backup and restore

• journaling must be enabled on the logical volume.

To backup, start by using command mongodump.

jason@localhost:~$ mongodump
connected to: 127.0.0.1
2014-07-07T22:46:09.351+0800 all dbs
2014-07-07T22:46:09.352+0800 DATABASE: test	 to 	dump/test
2014-07-07T22:46:09.354+0800 	test.system.indexes to dump/test/system.indexes.bson
2014-07-07T22:46:09.355+0800 		 4 documents
2014-07-07T22:46:09.356+0800 	test.testData to dump/test/testData.bson
2014-07-07T22:46:09.359+0800 		 400 documents
2014-07-07T22:46:09.360+0800 	Metadata for test.testData to dump/test/testData.metadata.json
2014-07-07T22:46:09.360+0800 	test.users to dump/test/users.bson
2014-07-07T22:46:09.361+0800 		 1 documents
2014-07-07T22:46:09.362+0800 	Metadata for test.users to dump/test/users.metadata.json
2014-07-07T22:46:09.362+0800 	test.accounts to dump/test/accounts.bson
2014-07-07T22:46:09.369+0800 		 2 documents
2014-07-07T22:46:09.370+0800 	Metadata for test.accounts to dump/test/accounts.metadata.json
2014-07-07T22:46:09.370+0800 	test.transactions to dump/test/transactions.bson
2014-07-07T22:46:09.372+0800 		 1 documents
2014-07-07T22:46:09.373+0800 	Metadata for test.transactions to dump/test/transactions.metadata.json
2014-07-07T22:46:09.374+0800 DATABASE: mydb	 to 	dump/mydb
2014-07-07T22:46:09.375+0800 	mydb.system.indexes to dump/mydb/system.indexes.bson
2014-07-07T22:46:09.376+0800 		 2 documents
2014-07-07T22:46:09.377+0800 	mydb.testData to dump/mydb/testData.bson
2014-07-07T22:46:09.378+0800 		 27 documents
2014-07-07T22:46:09.389+0800 	Metadata for mydb.testData to dump/mydb/testData.metadata.json
2014-07-07T22:46:09.390+0800 	mydb.users to dump/mydb/users.bson
2014-07-07T22:46:09.391+0800 		 1 documents
2014-07-07T22:46:09.392+0800 	Metadata for mydb.users to dump/mydb/users.metadata.json
2014-07-07T22:46:09.392+0800 DATABASE: mp3db	 to 	dump/mp3db
2014-07-07T22:46:09.393+0800 	mp3db.system.indexes to dump/mp3db/system.indexes.bson
2014-07-07T22:46:09.394+0800 		 4 documents
2014-07-07T22:46:09.395+0800 	mp3db.mp3.files to dump/mp3db/mp3.files.bson
2014-07-07T22:46:09.396+0800 		 1 documents
2014-07-07T22:46:09.397+0800 	Metadata for mp3db.mp3.files to dump/mp3db/mp3.files.metadata.json
2014-07-07T22:46:09.397+0800 	mp3db.mp3.chunks to dump/mp3db/mp3.chunks.bson
2014-07-07T22:46:09.401+0800 		 2 documents
2014-07-07T22:46:09.401+0800 	Metadata for mp3db.mp3.chunks to dump/mp3db/mp3.chunks.metadata.json
2014-07-07T22:46:09.402+0800 DATABASE: admin	 to 	dump/admin
2014-07-07T22:46:09.406+0800 DATABASE: config	 to 	dump/config

Let's remove some data from the database before we restore.

jason@localhost:~$ mongo
MongoDB shell version: 2.6.3
connecting to: test
Server has startup warnings:
2014-06-24T21:23:40.227+0800 [initandlisten]
2014-06-24T21:23:40.227+0800 [initandlisten] ** NOTE: This is a 32 bit MongoDB binary.
2014-06-24T21:23:40.227+0800 [initandlisten] **       32 bit builds are limited to less than 2GB of data (or less with --journal).
2014-06-24T21:23:40.227+0800 [initandlisten] **       Note that journaling defaults to off for 32 bit and is currently off.
2014-06-24T21:23:40.228+0800 [initandlisten] **       See http://dochub.mongodb.org/core/32bit
2014-06-24T21:23:40.228+0800 [initandlisten]
> use mp3db;
switched to db mp3db
> show tables;
mp3.chunks
mp3.files
system.indexes
> db.mp3.chunks.remove({});
WriteResult({ "nRemoved" : 2 })
> db.mp3.files.remove({});
WriteResult({ "nRemoved" : 1 })
> db.mp3.chunks.find();
> db.mp3.files.find();
>

Now we restore using command mongorestore.

jason@localhost:~$ mongorestore --collection mp3.chunks --db mp3db dump/mp3db/mp3.chunks.bson
connected to: 127.0.0.1
2014-07-07T23:14:43.504+0800 dump/mp3db/mp3.chunks.bson
2014-07-07T23:14:43.504+0800 	going into namespace [mp3db.mp3.chunks]
Restoring to mp3db.mp3.chunks without dropping. Restored data will be inserted without raising errors; check your server log
2 objects found
2014-07-07T23:14:43.534+0800 	Creating index: { key: { _id: 1 }, name: "_id_", ns: "mp3db.mp3.chunks" }
2014-07-07T23:14:43.635+0800 	Creating index: { key: { files_id: 1, n: 1 }, name: "files_id_1_n_1", ns: "mp3db.mp3.chunks" }

jason@localhost:~$ mongorestore --collection mp3.files --db mp3db dump/mp3db/mp3.files.bson
connected to: 127.0.0.1
2014-07-07T23:17:24.813+0800 dump/mp3db/mp3.files.bson
2014-07-07T23:17:24.813+0800 	going into namespace [mp3db.mp3.files]
Restoring to mp3db.mp3.files without dropping. Restored data will be inserted without raising errors; check your server log
1 objects found
2014-07-07T23:17:24.819+0800 	Creating index: { key: { _id: 1 }, name: "_id_", ns: "mp3db.mp3.files" }
2014-07-07T23:17:24.822+0800 	Creating index: { key: { filename: 1, uploadDate: 1 }, name: "filename_1_uploadDate_1", ns: "mp3db.mp3.files" }

Looks good, the restoration process and now we verify the content.

jason@localhost:~$ mongo
MongoDB shell version: 2.6.3
connecting to: test
Server has startup warnings:
2014-06-24T21:23:40.227+0800 [initandlisten]
2014-06-24T21:23:40.227+0800 [initandlisten] ** NOTE: This is a 32 bit MongoDB binary.
2014-06-24T21:23:40.227+0800 [initandlisten] **       32 bit builds are limited to less than 2GB of data (or less with --journal).
2014-06-24T21:23:40.227+0800 [initandlisten] **       Note that journaling defaults to off for 32 bit and is currently off.
2014-06-24T21:23:40.228+0800 [initandlisten] **       See http://dochub.mongodb.org/core/32bit
2014-06-24T21:23:40.228+0800 [initandlisten]
> use mp3db;
switched to db mp3db
> db.mp3.files.find();
{ "_id" : ObjectId("53ad61c844ae8a6ee12fcb63"), "chunkSize" : NumberLong(262144), "length" : NumberLong(316773), "md5" : "7293e9fd795e2bb6d5035e5b69cb2923", "filename" : "django.mp3", "contentType" : "audio/mpeg", "uploadDate" : ISODate("2014-06-27T12:21:28.646Z"), "aliases" : null }
>

Looks good. Now we move on to monitoring.

monitoring

mongostats - captures and returns the counts of database operations by type (e.g. insert, query, update, delete, etc.). These counts report on the load distribution on the server.

jason@localhost:~$ mongostat
connected to: 127.0.0.1
insert  query update delete getmore command flushes mapped  vsize    res faults   locked db idx miss %     qr|qw   ar|aw  netIn netOut  conn       time
    *0     *0     *0     *0       0     1|0       0   320m   445m    10m     14 config:0.0%          0       0|0     0|0    62b     3k     1   22:33:54
    *0     *0     *0     *0       0     1|0       0   320m   445m    10m      0   test:0.0%          0       0|0     0|0    62b     3k     1   22:33:55
    *0     *0     *0     *0       0     1|0       0   320m   445m    10m      0   test:0.0%          0       0|0     0|0    62b     3k     1   22:33:56
^C

mongotop tracks and reports the current read and write activity of a MongoDB instance, and reports these statistics on a per collection basis.

jason@localhost:~$ mongotop
connected to: 127.0.0.1

                            ns       total        read       write		2014-07-07T15:20:38
              mp3db.mp3.chunks         0ms         0ms         0ms
          local.system.replset         0ms         0ms         0ms
       local.system.namespaces         0ms         0ms         0ms
          local.system.indexes         0ms         0ms         0ms
             local.startup_log         0ms         0ms         0ms
                config.version         0ms         0ms         0ms
      config.system.namespaces         0ms         0ms         0ms

                            ns       total        read       write		2014-07-07T15:20:39
              mp3db.mp3.chunks         0ms         0ms         0ms
          local.system.replset         0ms         0ms         0ms
       local.system.namespaces         0ms         0ms         0ms
          local.system.indexes         0ms         0ms         0ms
             local.startup_log         0ms         0ms         0ms
                config.version         0ms         0ms         0ms
      config.system.namespaces         0ms         0ms         0ms
^C
jason@localhost:~$

HTTP Console - MongoDB provides a web interface that exposes diagnostic and monitoring information in a simple web page. For example , by accessing http://192.168.0.2:27017/

Now using db.serverStatus() from the mongo shell. The serverStatus command, or db.serverStatus() from the shell, returns a general overview of the status of the database, detailing disk usage, memory use, connection, journaling, and index access. The command returns quickly and does not impact MongoDB performance.

> db.serverStatus()
{
	"host" : "debby.e2e.serveftp.net",
	"version" : "2.6.3",
	"process" : "mongod",
	"pid" : NumberLong(3651),
	"uptime" : 1130615,
	"uptimeMillis" : NumberLong(1130614302),
	"uptimeEstimate" : 1115929,
	"localTime" : ISODate("2014-07-07T15:27:14.416Z"),
	"asserts" : {
		"regular" : 0,
		"warning" : 0,
		"msg" : 0,
		"user" : 2,
		"rollovers" : 0
	},
	"backgroundFlushing" : {
		"flushes" : 18843,
		"total_ms" : 127648,
		"average_ms" : 6.774292840842754,
		"last_ms" : 2,
		"last_finished" : ISODate("2014-07-07T15:26:43.282Z")
	},
	"connections" : {
		"current" : 1,
		"available" : 51199,
		"totalCreated" : NumberLong(57)
	},
	"cursors" : {
		"note" : "deprecated, use server status metrics",
		"clientCursors_size" : 0,
		"totalOpen" : 0,
		"pinned" : 0,
		"totalNoTimeout" : 2,
		"timedOut" : 1
	},
	"extra_info" : {
		"note" : "fields vary by platform",
		"heap_usage_bytes" : 23483832,
		"page_faults" : 13958
	},
	"globalLock" : {
		"totalTime" : NumberLong("1130614310000"),
		"lockTime" : NumberLong(184448),
		"currentQueue" : {
			"total" : 0,
			"readers" : 0,
			"writers" : 0
		},
		"activeClients" : {
			"total" : 0,
			"readers" : 0,
			"writers" : 0
		}
	},
	"indexCounters" : {
		"accesses" : 451,
		"hits" : 451,
		"misses" : 0,
		"resets" : 0,
		"missRatio" : 0
	},
	"locks" : {
		"." : {
			"timeLockedMicros" : {
				"R" : NumberLong(149),
				"W" : NumberLong(184448)
			},
			"timeAcquiringMicros" : {
				"R" : NumberLong(29),
				"W" : NumberLong(32)
			}
		},
		"admin" : {
			"timeLockedMicros" : {
				"r" : NumberLong(7348709),
				"w" : NumberLong(0)
			},
			"timeAcquiringMicros" : {
				"r" : NumberLong(55635),
				"w" : NumberLong(0)
			}
		},
		"local" : {
			"timeLockedMicros" : {
				"r" : NumberLong(59492773),
				"w" : NumberLong(32)
			},
			"timeAcquiringMicros" : {
				"r" : NumberLong(3164744),
				"w" : NumberLong(3)
			}
		},
		"config" : {
			"timeLockedMicros" : {
				"r" : NumberLong(182516),
				"w" : NumberLong(0)
			},
			"timeAcquiringMicros" : {
				"r" : NumberLong(46473),
				"w" : NumberLong(0)
			}
		},
		"mydb" : {
			"timeLockedMicros" : {
				"r" : NumberLong(43791920),
				"w" : NumberLong(118)
			},
			"timeAcquiringMicros" : {
				"r" : NumberLong(2159715),
				"w" : NumberLong(9)
			}
		},
		"test" : {
			"timeLockedMicros" : {
				"r" : NumberLong(28235652),
				"w" : NumberLong(252)
			},
			"timeAcquiringMicros" : {
				"r" : NumberLong(4052053),
				"w" : NumberLong(19)
			}
		},
		"mp3db" : {
			"timeLockedMicros" : {
				"r" : NumberLong(42491162),
				"w" : NumberLong(1053565)
			},
			"timeAcquiringMicros" : {
				"r" : NumberLong(6120501),
				"w" : NumberLong(832)
			}
		}
	},
	"network" : {
		"bytesIn" : 13516862,
		"bytesOut" : 34014948,
		"numRequests" : 733
	},
	"opcounters" : {
		"insert" : 112,
		"query" : 100247,
		"update" : 0,
		"delete" : 18,
		"getmore" : 3,
		"command" : 344
	},
	"opcountersRepl" : {
		"insert" : 0,
		"query" : 0,
		"update" : 0,
		"delete" : 0,
		"getmore" : 0,
		"command" : 0
	},
	"recordStats" : {
		"accessesNotInMemory" : 10,
		"pageFaultExceptionsThrown" : 1,
		"admin" : {
			"accessesNotInMemory" : 0,
			"pageFaultExceptionsThrown" : 0
		},
		"config" : {
			"accessesNotInMemory" : 0,
			"pageFaultExceptionsThrown" : 0
		},
		"local" : {
			"accessesNotInMemory" : 1,
			"pageFaultExceptionsThrown" : 0
		},
		"mp3db" : {
			"accessesNotInMemory" : 1,
			"pageFaultExceptionsThrown" : 1
		},
		"mydb" : {
			"accessesNotInMemory" : 2,
			"pageFaultExceptionsThrown" : 0
		},
		"test" : {
			"accessesNotInMemory" : 6,
			"pageFaultExceptionsThrown" : 0
		}
	},
	"writeBacksQueued" : false,
	"mem" : {
		"bits" : 32,
		"resident" : 12,
		"virtual" : 445,
		"supported" : true,
		"mapped" : 320
	},
	"metrics" : {
		"cursor" : {
			"timedOut" : NumberLong(1),
			"open" : {
				"noTimeout" : NumberLong(2),
				"pinned" : NumberLong(0),
				"total" : NumberLong(0)
			}
		},
		"document" : {
			"deleted" : NumberLong(72),
			"inserted" : NumberLong(112),
			"returned" : NumberLong(1294),
			"updated" : NumberLong(0)
		},
		"getLastError" : {
			"wtime" : {
				"num" : 0,
				"totalMillis" : 0
			},
			"wtimeouts" : NumberLong(0)
		},
		"operation" : {
			"fastmod" : NumberLong(0),
			"idhack" : NumberLong(0),
			"scanAndOrder" : NumberLong(0)
		},
		"queryExecutor" : {
			"scanned" : NumberLong(106),
			"scannedObjects" : NumberLong(106)
		},
		"record" : {
			"moves" : NumberLong(0)
		},
		"repl" : {
			"apply" : {
				"batches" : {
					"num" : 0,
					"totalMillis" : 0
				},
				"ops" : NumberLong(0)
			},
			"buffer" : {
				"count" : NumberLong(0),
				"maxSizeBytes" : 268435456,
				"sizeBytes" : NumberLong(0)
			},
			"network" : {
				"bytes" : NumberLong(0),
				"getmores" : {
					"num" : 0,
					"totalMillis" : 0
				},
				"ops" : NumberLong(0),
				"readersCreated" : NumberLong(0)
			},
			"preload" : {
				"docs" : {
					"num" : 0,
					"totalMillis" : 0
				},
				"indexes" : {
					"num" : 0,
					"totalMillis" : 0
				}
			}
		},
		"storage" : {
			"freelist" : {
				"search" : {
					"bucketExhausted" : NumberLong(0),
					"requests" : NumberLong(97),
					"scanned" : NumberLong(166)
				}
			}
		},
		"ttl" : {
			"deletedDocuments" : NumberLong(0),
			"passes" : NumberLong(18840)
		}
	},
	"ok" : 1
}
>

dbStats - The dbStats command, or db.stats() from the shell, returns a document that addresses storage use and data volumes. The dbStats reflect the amount of storage used, the quantity of data contained in the database, and object, collection, and index counters.

> use mp3db
switched to db mp3db
> db.stats()
{
	"db" : "mp3db",
	"collections" : 4,
	"objects" : 14,
	"avgObjSize" : 42210.28571428572,
	"dataSize" : 590944,
	"storageSize" : 35037184,
	"numExtents" : 7,
	"indexes" : 4,
	"indexSize" : 32704,
	"fileSize" : 67108864,
	"nsSizeMB" : 16,
	"dataFileVersion" : {
		"major" : 4,
		"minor" : 5
	},
	"extentFreeList" : {
		"num" : 0,
		"totalSize" : 0
	},
	"ok" : 1
}
>

collStats - The collStats provides statistics that resemble dbStats on the collection level, including a count of the objects in the collection, the size of the collection, the amount of disk space used by the collection, and information about its indexes.

> db.mp3.chunks.stats()
{
	"ns" : "mp3db.mp3.chunks",
	"count" : 2,
	"size" : 589792,
	"avgObjSize" : 294896,
	"storageSize" : 35012608,
	"numExtents" : 4,
	"nindexes" : 2,
	"lastExtentSize" : 15290368,
	"paddingFactor" : 1,
	"systemFlags" : 1,
	"userFlags" : 1,
	"totalIndexSize" : 16352,
	"indexSizes" : {
		"_id_" : 8176,
		"files_id_1_n_1" : 8176
	},
	"ok" : 1
}
>

replSetGetStatus - The replSetGetStatus command (rs.status() from the shell) returns an overview of your replica set’s status. The replSetGetStatus document details the state and configuration of the replica set and statistics about its members.

> rs.status()
{ "ok" : 0, "errmsg" : "not running with --replSet" }

log available in /var/log/mongodb/mongod.log

configuration

There are too many configurations to covered here but below are the essential configurations which you might need to change.

As mentioned previously, if the app that connected to the database are on two different servers, then in the server that run mongo instance, you should comment out bind_ip

# Listen to local interface only. Comment out to listen on all interfaces.
#bind_ip = 127.0.0.1

Default port running is 27017 but you should make sure that this port and ip allow to be access remotely.

use kernel 2.6.36 or later.

In general, if you use the Ext4 file system, use at least version 2.6.23 of the Linux Kernel.

In general, if you use the XFS file system, use at least version 2.6.25 of the Linux Kernel.

Set the file descriptor limit, -n, and the user process limit (ulimit), -u, above 20,000, according to the suggestions in the ulimit document. A low ulimit will affect MongoDB when under heavy use and can produce errors and lead to failed connections to MongoDB processes and loss of service.

That's it. Please go to the donate page and contribute back if you learned something.

CentOS 7 released and checking out release notes

I'm always a big fan and user of CentOS. Started using CentOS 4 and it is very stables and secure for company servers usage. With the current release of CentOS 7, it is definitely worth while to check in out the release note.

It is definitely encourage to see that, for the first time, major upgrade between major CentOS is possible now.

The install media is splited such that it comes with the window managers. For server installation, you should really go using net install.

These are some major changes which should be noticeable. Like Wow!

• Kernel updated to 3.10.0


• Support for Linux Containers


• Open VMware Tools and 3D graphics drivers out of the box


• OpenJDK-7 as default JDK


• In Place Upgrade from 6.5 to 7.0 (as already mentioned)


• Switch to systemd, firewalld and GRUB2


• XFS as default file system


• iSCSI and FCoE in kernel space


• Support for PTPv2


• Support for 40G Ethernet Cards


• Supports installations in UEFI Secure Boot mode on compatible hardware

There are some known issues which you should really consider if you are doing upgrade and make sure you are well prepare.
network - Many people have complained that Ethernet interfaces are not started with the new default NetworkManager tool/have to be explicitly enabled during installation. See CentOS-7 FAQ#2.
installer memory usage - The installer needs at least 406MB of memory to work. On systems with less memory then 406MB the installation will terminate with a fatal error. 512MB is the minimum memory requirement for CentOS-7.
small screen - If your screen resolution is 800x600 or lower, parts of the images shown at the bottom during install are clipped. So watch up on the next,back and cancel buttons.
So I think this is a brief summary to get you started but you can find more at here.

Cassandra discarding obsolete commit log

Often times, I noticed the log show the following lines in apache cassandra 1.0.8

INFO [COMMIT-LOG-WRITER] 2014-06-30 08:58:55,039 CommitLog.java (line 490) Discarding obsolete commit log:CommitLogSegment(/mnt/cassandra/commitlog/CommitLog-1404095354976.log)

It looks nothing to worry about as the log level is INFO. However, I often see this and just to check to understand what it really is.

This log is written by CommitLog.java.

private void maybeDiscardSegment(CommitLogSegment segment, Iterator<CommitLogSegment> iter)
{
    if (segment.isSafeToDelete() && iter.hasNext())
    {
        logger.info("Discarding obsolete commit log:" + segment);
        FileUtils.deleteAsync(segment.getPath());
        // usually this will be the first (remaining) segment, but not always, if segment A contains
        // writes to a CF that is unflushed but is followed by segment B whose CFs are all flushed.
        iter.remove();
    }
    else
    {
        if (logger.isDebugEnabled())
            logger.debug("Not safe to delete commit log " + segment + "; dirty is " + segment.dirtyString() + "; hasNext: " + iter.hasNext());
    }
}

So why discard segment? From the code documentation

Commit Log tracks every write operation into the system. The aim of the commit log is to be able to successfully recover data that was not stored to disk via the Memtable. Every Commit Log maintains a header represented by the abstraction CommitLogHeader. The header contains a bit array and an array of longs and both the arrays are of size, #column families for the Table, the Commit Log represents.

Whenever a ColumnFamily is written to, for the first time its bit flag is set to one in the CommitLogHeader. When it is flushed to disk by the Memtable its corresponding bit in the header is set to zero. This helps track which CommitLogs can be thrown away as a result of Memtable flushes. Additionally, when a ColumnFamily is flushed and written to disk, its entry in the array of longs is updated with the offset in the Commit Log file where it was written. This helps speed up recovery since we can seek to these offsets and start processing the commit log.

Every Commit Log is rolled over everytime it reaches its threshold in size; the new log inherits the "dirty" bits from the old.

Over time there could be a number of commit logs that would be generated. To allow cleaning up non-active commit logs, whenever we flush a column family and update its bit flag in the active CL, we take the dirty bit array and bitwise & it with the headers of the older logs. If the result is 0, then it is safe to remove the older file. (Since the new CL inherited the old's dirty bitflags, getting a zero for any given bit in the anding means that either the CF was clean in the old CL or it has been flushed since the switch in the new.)

So that's pretty clear why commit log is remove. Tracing the call upward,

CommitLog.maybeDiscardSegment()
  ^
  |
  +--- CommitLog.discardCompletedSegmentsInternal()
    ^
    |
    +--- CommitLog.discardCompletedSegments()
       ^
       |
       +--- ColumnFamilyStore.maybeSwitchMemtable()
       |
       +--- ColumnFamilyStore.truncate()

So whenever a column family is truncated, the commit log is discard (remove) as well which make sense. When maybeSwitchMemtable is executed, that is, memtable is flushed, after that, its segments get discard (remove) as well.

This logging message is just fine as it is part of cassandra operation. That's it for this article.

Retro Dude Review: Pitfall

Hello everyone! Shall we play a game?

This time we're gonna take a closer look on a classic atari 2600 game better known as Pitfall or Pitfall Harry's Jungle Adventure.

The year was 1982 and the gaming industri was booming. An arcade on every corner and Starcade on the T.V. not a bad year to be a gamer at all. Atari was making the big bucks but failed to give credit where credit was due and because of that Activision was born. But thats enough brief history for now.

Pitfall made by brilliant designer David Crane who is also responsible for such titles as Dragster, Freeway, Laser Blast and many others, is an adventure game where you take the role as Harry. Your task is to find the 32 treasures of the jungle within 20 minutes while avoiding pits, lakes, crokodiles, snakes, huge scorpions, rolling logs everywhere and much more. The game ends when all the treasure is found, your time is up or when you have used up all your lives. The game itself contains 255 trees all of wich Harry will pass on his journey making the game a cirkular maze. That basicly means when you have finished the end screen the first screen reappears. Now while the graphics might not seem like much today back in that time it must have been impressive I'm sure. The sounds are typical Atari and it's the first game I have ever encountered with no music. There is only sound while you jump, die or run into logs and it makes the game at first glance seem rather boring (But I guess it beats the Astroids soundtrack) The gameplay on the other hand is very nice. Its smooth controlling all the way but its still hard as *!#@ Trust me on this one when you try jumping over a crokodile pit your accuracy and precision has to be perfect, and I do mean perfect if you're just a split second off you are dead. Then we got these almost trap like pits that just apears out of nowhere on the ground and they will lure you into sweet, sweet game over, right until you're ready to explode of pure agony! But once you get the hang of it the only real trick is to collect all the treasure before the timer runs out. So far after countless tries im still missing 2 (But hey im no expert)



Pitfall is all in all a very decent game compared to alot of other games that came out in that time period, the graphics and gameplay is nice and its almost addictive when you decide on getting all 32 treasures, and its one of the few atari games with an actual end goal, not just scoring points for all eternity. The commercial even had a very young Jack Black (that  at the time might not have meant that much but today its pretty damn cool) So all in all if I had to rate it I would say its an 8/10 due to the missing soundtrack.

 

Thats all for my first review this is the Retro Dude saying goodbye and I do hope you get a chance on playing an instant classic!

Learning java selenium webdriver

Today, we are going to something different, we explore software test technology. As end user, we are mostly dealing with point and click, and with this article, we are going to learn how to test user interface.

There are many testing software out there, just google and you find plenty. In this article, we are going to use Java Selenium webdriver. To quickly start, you will have to complete these bullet points. For complete information of selenium, read here.

• download latest stable selenium java driver


• setup your IDE to reference this jar.

or additional optional download at http://docs.seleniumhq.org/download/



Let's see continue this article with two example.

import java.util.List;

import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.firefox.FirefoxDriver;

public class GoogleSuggest {

    public static void main(String[] args) throws Exception {
        // The Firefox driver supports javascript
        WebDriver driver = new FirefoxDriver();

        // Go to the Google Suggest home page
        driver.get("http://www.google.com/webhp?complete=1&hl=en");

        // Enter the query string "Cheese"
        WebElement query = driver.findElement(By.name("q"));
        query.sendKeys("Cheese");

        // Sleep until the div we want is visible or 5 seconds is over
        long end = System.currentTimeMillis() + 5000;
        while (System.currentTimeMillis() < end) {
            WebElement resultsDiv = driver.findElement(By.className("gssb_e"));

            // If results have been returned, the results are displayed in a drop down.
            if (resultsDiv.isDisplayed()) {
              break;
            }
        }

        // And now list the suggestions
        List<WebElement> allSuggestions = driver.findElements(By.xpath("//td[@class='gssb_a gbqfsf']"));

        for (WebElement suggestion : allSuggestions) {
            System.out.println("suggestion => " + suggestion.getText());
        }

        driver.quit();
    }

}

import org.junit.Before;
import org.junit.Test;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.firefox.FirefoxDriver;

import com.thoughtworks.selenium.SeleneseTestBase;
import com.thoughtworks.selenium.webdriven.WebDriverBackedSelenium;

public class BaseSeleniumTest extends SeleneseTestBase {

    @Before
    public void setUp() throws Exception {
            WebDriver driver1 = new FirefoxDriver();
            String baseUrl = "http://google.com/";
            selenium = new WebDriverBackedSelenium(driver1, baseUrl);
    }

    @Test
    public void testGoogle() throws Exception {
    	selenium.open("http://google.com/");
    	System.out.println(selenium.getTitle());
    	assertEquals("Google", selenium.getTitle());
    	selenium.close();
    }
}

The first example construct a FireFox WebDriver. Load google url and mimic a typing by sending a keys of Cheese to the query. When Web Element resultsDiv is displayed, then all the suggestion get printed.

The second example is much easier to construct test. By extending SeleneseTestBase, you can use the object selenium and start to call the rich methods it provides for your sub test class. As seem here, similar to the first example, the test started with construction of FirefoxDriver. Then the very familiar junit test, open up google link, assert that the title is Google and then close the selenium object.

That's it for this article, I hope it help you get started. Remember to donate us if you would like to contribute back.

Study MongoDB GridFS with java example

In the past, we have learned basic MongoDB and study data model, in this article , we will study MongoDB GridFS by storing a file into MongoDB. Below is the java simple application to show how to store, retrieve, and delete eventually.

import java.io.File;
import java.io.IOException;

import com.mongodb.DB;
import com.mongodb.DBCursor;
import com.mongodb.Mongo;
import com.mongodb.MongoException;
import com.mongodb.gridfs.GridFS;
import com.mongodb.gridfs.GridFSDBFile;
import com.mongodb.gridfs.GridFSInputFile;

public class LearnMongo {

	public static void main(String[] args) throws MongoException, IOException {
		Mongo mongo = new Mongo("192.168.0.2", 27017);
		DB db = mongo.getDB("mp3db");

		// save image
		String newFilename = "django.mp3";
		File mp3File = new File("src/resources/django.mp3");
		GridFS gfsMp3 = new GridFS(db, "mp3");
		GridFSInputFile gfsFile = gfsMp3.createFile(mp3File);
		gfsFile.setFilename(newFilename);
		gfsFile.setContentType("audio/mpeg");
		System.out.println(gfsFile.toString());
		gfsFile.save();

		// get mp3
		GridFSDBFile imageForOutput = gfsMp3.findOne(newFilename);
		System.out.println(imageForOutput);

		// print image
		DBCursor cursor = gfsMp3.getFileList();
		while (cursor.hasNext()) {
			System.out.println(cursor.next());
		}

		// save into another image
		imageForOutput.writeTo("/home/jason/Desktop/newsong.mp3");

		// delete image
		gfsMp3.remove(gfsMp3.findOne(newFilename));

	}

}

We start by connecting to the server, so with this example MongoDB instance running on server 192.168.0.2 on port 27017. You may want to check the configuration for MongoDB if you connect remotely as the default MongoDB configuration only listen to localhost.

Then we form a MongoDB DB object on mp3db. You can store other object as well but for this example, I'm going to store a mp3. With this ready, we are going to store the mp3. The important piece of code is probably below.

GridFS gfsMp3 = new GridFS(db, "mp3");
GridFSInputFile gfsFile = gfsMp3.createFile(mp3File);

Instantiate two object, GrisFS and GridFSInputFile. You can set additional information like filename, content type. Calling GridFSInputFile.save() will save the object into MongoDB. If you have access to MongoDB cli, command such as > db.mp3.files.find(); will shown below the output.

{ "_id" : ObjectId("53ad60f944aeaca83109d253"), "chunkSize" : NumberLong(262144), "length" : NumberLong(316773), "md5" : "7293e9fd795e2bb6d5035e5b69cb2923", "filename" : "django.mp3", "contentType" : "audio/mpeg", "uploadDate" : ISODate("2014-06-27T12:18:01.934Z"), "aliases" : null }

To find the mp3, you can use the code, GridFSDBFile imageForOutput = gfsMp3.findOne(newFilename); below is the output.

{ "_id" : { "$oid" : "53ad60f944aeaca83109d253"} , "chunkSize" : 262144 , "length" : 316773 , "md5" : "7293e9fd795e2bb6d5035e5b69cb2923" , "filename" : "django.mp3" , "contentType" : "audio/mpeg" , "uploadDate" : { "$date" : "2014-06-27T12:18:01Z"} , "aliases" : null }

You can also use GridFS.getFileList(); to retrieve all the files currently store on this database. The code continue on writing the object into a file. As you can see, I'm writing to desktop just to ensure it is not from the source.

I end this article by removing the object in the MongoDB database.

Study MongoDB data models

Today we are going to learn on MongoDB Data Models.

It is important to study data modal is because as a developer, you would want to leverage what MongoDB is excel at and aware what it is not suitable for.

You can basically store a few document and reference then using and id field.
But remember this need two round trip back and forth from the application servers
to the mongo database.

As such, in this scenario if it better to embed the document into a document.

user document
{
   _id: <ObjectId1>,      <-------------+
   username : "jasonwee"                |
}                                       |
                                        |
contact document                        |
{                                       |
   _id: <ObjectId2>,                    |
   user_id: <ObjectId1>   <-------------+
   phone: "012-3456789"
}

into

user document
{
   _id: <ObjectId1>,
   contact : {
      phone: "012-3456789"
   }
}

This modelling guarantee you atomicity of a document as mongodb write operations
are atomic at document level.

Indexes

Use indexes to improve performance for common queries. Build indexes on fields that appear often in queries and for all operations that return sorted results. MongoDB automatically creates a unique index on the _id field.

Each index requires at least 8KB of data space.

GridFS

GridFS is a specification for storing and retrieving files that exceed the BSON-document size limit of 16MB.

Model Relationships Between Documents

• Model One-to-One Relationships with Embedded Documents
  Presents a data model that uses embedded documents to describe one-to-one relationships between connected data.


• Model One-to-Many Relationships with Embedded Documents
  Presents a data model that uses embedded documents to describe one-to-many relationships between connected data.


• Model One-to-Many Relationships with Document References
  Presents a data model that uses references to describe one-to-many relationships between documents.

Model Tree Structures

MongoDB allows various ways to use tree data structures to model large hierarchical or nested data relationships.

• Model Tree Structures with Parent References 
  Presents a data model that organizes documents in a tree-like structure by storing references to “parent” nodes in “child” nodes.


• Model Tree Structures with Child References
  Presents a data model that organizes documents in a tree-like structure by storing references to “child” nodes in “parent” nodes.


• Model Tree Structures with an Array of Ancestors
  Presents a data model that organizes documents in a tree-like structure by storing references to “parent” nodes and an array that stores all ancestors.


• Model Tree Structures with Materialized Paths
  Presents a data model that organizes documents in a tree-like structure by storing full relationship paths between documents. In addition to the tree node, each document stores the _id of the nodes ancestors or path as a string.


• Model Tree Structures with Nested Sets
  Presents a data model that organizes documents in a tree-like structure using the Nested Sets pattern. This optimizes discovering subtrees at the expense of tree mutability.