About

The eLog library was initially developed as a research prototype and later published for lifelogging researchers in 2010 to help them easily analyze heterogenous data and complex visualization. It has been kept growing with the progress of mobile computing environments and recently its UI part is released with GPL v3 license for wider usage. The eLog UI library is optimized for mobile environment and can get easily integrated with existing Web services.

Who We Are

The original work was proposed by Pil Ho and later extended the work with collaboration with 28 researchers around the world who contributed their lifelogs, collaborated for lifelog analysis and share research results to build up an open lifelogging platform for the public. Pil Ho has been keeping the development updating the library following up the progress in mobile computing.

Updates

  • Nov. 2014: Change the web page skin using bootstrap.
  • Nov. 2014: Published elog UI library as GPL v3.
  • Oct. 2014: Version up eLog library and documentation.

 

Microsoft GeoLife GPS Trajectories

GeoLife GPS Trajectories is a dataset collected in (Microsoft Research Asia) Geolife project by 182 users in a period of over three years (from April 2007 to August 2012). Recently they updated the data and the below is accordingly updated.

Data Description

This dataset is available at their download page. If you want to use 'wget' like tool, then check the source of this page to find the direct link to the file. File download size is 298.66 MB and it includes many PLT gps files and Labels.txt (Ground truth user transportation mode). More details on their data can be found at the included user guide in their download file.

Data Parsing and Database Preparation

We imported all into our MySQL database. Below is the sequence of commands to import and process all data. One difference with the old data is now that they have Labels.txt which is the ground-truth data tagged by users. So we redesigned tables like below.

Prepare MySQL Tables

We have two PLT tables: one for fast importing without indexing and the other for distinct row importing with the complete index. We may create a complete table at the first momemt but since it involves shell scripts and background process, let move the field to the MySQL, which is much easier and safer.

CREATE TABLE `plt` (
  `directory` varchar(10) DEFAULT NULL,
  `latitude` double DEFAULT NULL,
  `longitude` double DEFAULT NULL,
  `flag` int(11) DEFAULT NULL,
  `altitude` double DEFAULT NULL,
  `passeddate` varchar(255) DEFAULT NULL,
  `gpsdate` date DEFAULT NULL,
  `gpstime` time DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=latin1;

CREATE TABLE `plt_distinct` (
  `directory` varchar(10) NOT NULL DEFAULT '',
  `latitude` double NOT NULL DEFAULT '0',
  `longitude` double NOT NULL DEFAULT '0',
  `flag` int(11) DEFAULT NULL,
  `altitude` double NOT NULL DEFAULT '0',
  `passeddate` varchar(255) DEFAULT NULL,
  `gpsdate` date NOT NULL DEFAULT '0000-00-00',
  `gpstime` time NOT NULL DEFAULT '00:00:00',
  `gpsdatetime` timestamp NULL DEFAULT NULL,
  PRIMARY KEY (`directory`,`latitude`,`longitude`,`gpsdate`,`gpstime`,`altitude`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;

CREATE TABLE `labels` (
  `directory` varchar(10) NOT NULL DEFAULT '',
  `starttime` datetime NOT NULL DEFAULT '0000-00-00 00:00:00',
  `endtime` datetime NOT NULL DEFAULT '0000-00-00 00:00:00',
  `transportationmode` varchar(10) NOT NULL DEFAULT '',
  PRIMARY KEY (`directory`,`starttime`,`endtime`,`transportationmode`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;

Importing data

Unlike old one below, let's do everything using shell scripts not PHP. Belows are BASH schell scripts. If you are using Windows, please find similar tools or use cygwin.

For both scripts below, first open the shell and move to the unzipped data directory (ex. GeolifeTrajectories1.3/Data) and then create script files using below script (Don't forget to set the execution attribute!)

#!/bin/bash                                                                     

shopt -s globstar

for file in **/*.plt
do
    if [[ ! -f "$file" ]]
    then
        continue
    fi  
    echo "$file"
    # Do something else.                                                          
    
    rm plt.txt
    ln -s "$file" "plt.txt"
    cols="latitude, longitude, @flag, altitude, passeddate, gpsdate, gpstime) SET directory = '${file:0:3}', flag=(@flag"
    mysqlimport -i -s --local --user=yourdbuserid --password=yourdbuserpassword --fields-terminated-by="," --lines-terminated-by="\r\n" --ignore-lines=6 --columns="$cols" userdbname plt.txt 
done

Below scripts import Labels.txt data from the GeoLife data set

 

#!/bin/bash                                                                     

shopt -s globstar

for file in **/labels.txt
do
    if [[ ! -f "$file" ]]
    then
        continue
    fi  
    echo "$file"
    # Do something else.                                                          
    
    rm labels.txt
    ln -s "$file" "labels.txt"
    cols="starttime, endtime, @transportationmode) SET directory = '${file:0:3}', transportationmode=(@transportationmode"
    
    mysqlimport --local --user=youruserid --password=yourpassword --fields-terminated-by="\t" --lines-terminated-by="\r\n" --ignore-lines=1 --columns="$cols" yourdbname labels.txt 
done

Once all steps are done (~< 1 hour), then execute MySQL client and check the data. If everything looks OK, then run below SQL commands.

 

INSERT IGNORE INTO plt_distinct (directory, latitude, longitude, flag, altitude, passeddate, gpsdate, gpstime, gpsdatetime)  SELECT directory, latitude, longitude, flag, altitude, passeddate, gpsdate, gpstime, CONCAT(gpsdate, ' ', gpstime) FROM plt;

As a statistics, there were total: 14718 labels and 5,405,681 GPS samples.

References

 

Obsolete

Note! Microsoft GeoLife GPS Trajectories data is changed and the below information is only compatible with their old (first version) data set.

This lab is to import Microsoft GeoLife GPS Trajectories data into the database.

Extract GPS trajectories

The dataset is composed of over two years (from April 2007 to August 2009) 165 users' GPS trajectories.

Data conversion for database import

Source GPS trajectories are in OziExplorer format (.plt extension). Its format is available at http://www.rus-roads.ru/gps/help_ozi/fileformats.html.

Table structure to store the plt format:

CREATE TABLE `geolife` (
  `gps_userid` int(11) DEFAULT NULL,
  `gps_latitude` double DEFAULT NULL,
  `gps_longitude` double DEFAULT NULL,
  `gps_code` int(11) DEFAULT NULL,
  `gps_altitude` double DEFAULT NULL,
  `gps_UTC_timestamp` timestamp NULL DEFAULT NULL,
  `gps_UTC_unix_timestamp` int(11) DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

PHP scripts to import data into the MySQL table:

query($query))) {
	printf("Table truncation error: %s\n", $mysqli->error);
	exit();
}

// Recursively read all files in the selected directory 
$ite=new RecursiveDirectoryIterator($path);
$bytestotal=0;
$nbfiles=0;
$gpspath = $path;
$gpssubpath = '';

$ite_gps = new RecursiveDirectoryIterator($gpspath);
foreach (new RecursiveIteratorIterator($ite_gps) as $gpsfilename=>$gpscur) {
	// Check sub path
	$newgpspath = $gpscur->getPathname();
	$newgpsfile = $gpscur->getFileName();
	$newgpsfullpath = substr($newgpspath, 0, strlen($newgpspath) - strlen($newgpsfile));
	$newgpssubpath = substr($newgpsfullpath, strlen($gpspath) - strlen($newgpsfullpath));

	if ($gpssubpath != $newgpssubpath && is_dir($gpspath.$newgpssubpath)) {  
		$gpssubpath = $newgpssubpath;
		echo "Entering the subpath: $gpssubpath\n";
	}

	if (strstr($gpscur, ".plt") != "") {
		$filesize=$gpscur->getSize();
		$modified_time = $gpscur->getMTime();
		$bytestotal+=$filesize;
		$nbfiles++;
		echo "$gpsfilename => $filesize\n";

		// Load data into the table
		$user_id = str_replace("/trajectory/", "", $gpssubpath);
		$query = " LOAD DATA LOCAL INFILE '".$gpscur->getPathname()."' ";
	  	$query .= " IGNORE INTO TABLE geolife ";
		$query .= " FIELDS TERMINATED BY ',' LINES TERMINATED BY '\\r\\n' IGNORE 6 LINES ";
		$query .= " (gps_latitude, gps_longitude, gps_code, gps_altitude, @gps_dateformat, @gps_date, @gps_time) ";
		$query .= " SET gps_userid = '".$user_id."', ";
		$query .= " gps_UTC_timestamp = CONCAT(@gps_date, ' ', @gps_time), ";
		$query .= " gps_UTC_unix_timestamp = UNIX_TIMESTAMP(CONCAT(@gps_date, ' ', @gps_time))";
		
		if (!($result = $mysqli->query($query))) {
			echo $query."\n";
			printf("GPS data insertion error: %s\n", $mysqli->error);
			return 0;
		}
	}
}	

unset($gpsfilename, $query);

/* Update the photo timestamp by the location */

$mysqli->close();

$bytestotal=number_format($bytestotal);
echo "Total: $nbfiles files, $bytestotal bytes\n";

?>

As a statistics, there were total: 8809 files in 1,554,806,869 bytes.

 

 

Copyrights

Any members using this data should conform to the following agreement:

 

Microsoft Research License Agreement

Non-Commercial Use Only << GeoLife GPS Trajectories>>
This Microsoft Research License Agreement, including all exhibits ("MSR-LA") is a legal agreement between you and Microsoft Corporation (Microsoft or we) for the software or data identified above, which may include source code, and any associated materials, text or speech files, associated media and "online" or electronic documentation and any updates we provide in our discretion (together, the "Software").
By installing, copying, or otherwise using this Software, you agree to be bound by the terms of this MSR-LA. If you do not agree, do not install copy or use the Software. The Software is protected by copyright and other intellectual property laws and is licensed, not sold.
SCOPE OF RIGHTS: You may use this Software for any non-commercial purpose, subject to the restrictions in this MSR-LA. Some
purposes which can be non-commercial are teaching, academic research, public demonstrations and personal experimentation. You may not distribute this Software or any derivative works in any form. In return, we simply require that you agree: 1. That you will not remove any copyright or other notices from the Software.
2. That if any of the Software is in binary format, you will not attempt to modify such portions of the Software, or to reverse engineer or decompile them, except and only to the extent authorized by applicable law. 3. That Microsoft is granted back, without any restrictions or limitations, a non-exclusive, perpetual, irrevocable, royalty-free, assignable and sub-licensable license, to reproduce, publicly perform or display, install, use, modify, post, distribute, make and have made, sell and transfer your modifications to and/or derivative works of the Software source code or data, for any purpose.
4. That any feedback about the Software provided by you to us is voluntarily given, and Microsoft shall be free to use the feedback as it sees fit without obligation or restriction of any kind, even if the feedback is designated by you as confidential. 5. THAT THE SOFTWARE COMES "AS IS", WITH NO WARRANTIES. THIS MEANS NO EXPRESS, IMPLIED OR STATUTORY WARRANTY, INCLUDING WITHOUT LIMITATION, WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE, ANY WARRANTY AGAINST INTERFERENCE WITH YOUR ENJOYMENT OF THE SOFTWARE OR ANY WARRANTY OF TITLE OR NON-INFRINGEMENT. THERE IS NO WARRANTY THAT THIS SOFTWARE WILL FULFILL ANY OF YOUR PARTICULAR PURPOSES OR NEEDS.
6. THAT NEITHER MICROSOFT NOR ANY CONTRIBUTOR TO THE SOFTWARE WILL BE LIABLE FOR ANY DAMAGES RELATED TO THE SOFTWARE OR THIS MSR-LA, INCLUDING DIRECT, INDIRECT, SPECIAL, CONSEQUENTIAL OR INCIDENTAL DAMAGES, TO THE MAXIMUM EXTENT THE LAW PERMITS, NO MATTER WHAT LEGAL THEORY IT IS BASED ON.
7. That we have no duty of reasonable care or lack of negligence, and we are not obligated to (and will not) provide technical support for the Software. 8. That if you breach this MSR-LA or if you sue anyone over patents that you think may apply to or read on the Software or anyone's use of the Software, this MSR-LA (and your license and rights obtained herein) terminate automatically. Upon any such termination, you shall destroy all of your copies of the Software immediately. Sections 3, 4, 5, 6, 7, 8, 11 and 12 of this MSR-LA shall survive any termination of this MSR-LA.
9. That the patent rights, if any, granted to you in this MSR-LA only apply to the Software, not to any derivative works you make. 10. That the Software may be subject to U.S. export jurisdiction at the time it is licensed to you, and it may be subject to additional export or import laws in other places. You agree to comply with all such laws and regulations that may apply to the Software after delivery of the software to you.
11. That all rights not expressly granted to you in this MSR-LA are reserved. 12. That this MSR-LA shall be construed and controlled by the laws of the State of Washington, USA, without regard to conflicts of law. If any provision of this MSR-LA shall be deemed unenforceable or contrary to law, the rest of this MSR-LA shall remain in full effect and interpreted in an enforceable manner that most nearly captures the intent of the original language.
Copyright (c) Microsoft Corporation. All rights reserved.