Monday, October 24, 2016

Tip to speed up the Android Studio project build compilation process

Android Studio builds Android projects with the default Gradle property settings. If you find the build process taking a long time and your computer has adequate memory, it is possible to tweak the settings to allocate more memory to the build process to reduce the compilation time.

In Android Studio, open up the top level gradle.properties file to change the settings, as shown below.


In the editor, locate the commented line with the string "org.gradle.jvmargs", and make a copy of it directly underneath; then remove the # character to un-comment it.

An example of an edited gradle.properties file is shown below.

# Project-wide Gradle settings.

# IDE (e.g. Android Studio) users:
# Gradle settings configured through the IDE *will override*
# any settings specified in this file.

# For more details on how to configure your build environment visit
# http://www.gradle.org/docs/current/userguide/build_environment.html

# Specifies the JVM arguments used for the daemon process.
# The setting is particularly useful for tweaking memory settings.
# Default value: -Xmx10248m -XX:MaxPermSize=256m
# org.gradle.jvmargs=-Xmx2048m -XX:MaxPermSize=512m -XX:+HeapDumpOnOutOfMemoryError -Dfile.encoding=UTF-8
org.gradle.jvmargs=-Xmx2048m -XX:MaxPermSize=512m -XX:+HeapDumpOnOutOfMemoryError -Dfile.encoding=UTF-8

# When configured, Gradle will run in incubating parallel mode.
# This option should only be used with decoupled projects. More details, visit
# http://www.gradle.org/docs/current/userguide/multi_project_builds.html#sec:decoupled_projects
# org.gradle.parallel=true

Now, the build process should complete faster.

Monday, October 17, 2016

Example efficient two pass Spatialite query to find if a point is inside a polygon

A spatial query can be expensive to run in terms of CPU processing power. In order to improve the query performance, typically, a first pass query is done to reduce the candidates first before running a second pass query to do actual spatial query. The following is an example using SpatiaLite to perform a two pass spatial query to find whether a point is contained within a polygon.
Point 1 is totally outside of any polygons; Point 2 is outside but within a bounding rectangle of a polygon; Point 3 is totally inside a polygon.
The two pass SpatiaLite query has the following syntax.
Note: the first approximate query uses the MBRCONTAINS function and the second finer query uses the ST_CONTAINS function.
SELECT ST_CONTAINS(polygonGeometry, geomfromtext('POINT(116.4688 -31.5502)', 4326))
FROM(
SELECT
polygonGeometry,
MBRCONTAINS(polygonGeometry, geomfromtext('POINT(116.4688 -31.5502)', 4326)) AS inMBR
FROM
(
SELECT geometry AS polygonGeometry FROM MyPolygon
)
WHERE inMBR = 1
)



Case 1: Totally outside 
For Point 1 (Totally outside), the query returns no rows since the point is outside of any bounding rectangles, as shown in the screenshot below.


Case 2: Outside of any polygons but within a bounding rectangle
For Point2, the query returns a result of 0 value, since the point is inside a bounding rectangle but outside of any polygon.

Case 3: Totally inside a polygon
For Point 3, the query returns a result of 1 since the point is inside a bounding rectangle and also inside a polygon.


Monday, October 10, 2016

Simple C# example for creating and using a Spatialite database

A Spatialite database is just a SQLite database loaded with Spatialite module extensions including tables, triggers, functions etc. It is therefore possible to access a Spatialite database using the SQLite .Net Wrapper from https://system.data.sqlite.org and with the Spatialite module extension library mod_spatialite-x.x.x from http://www.gaia-gis.it/gaia-sins/, otherwise the spatial functions will not be available to your C# code.

While the Spatialite database can be created using the SQLite wrapper and the Spatialite module extension, the newly created database lacks all the Spatialite tables, triggers etc. necessary for proper functioning of the Spatialite database. I would need to run some initialization SQL to create all the required Spatialite tables, triggers, etc. So instead, I found it more convenient to create a template Spatialite database with the Spatialite executable beforehand. Then in the C# program, just simple copy the template to create a new Spatialite database.

 The following is an example C# code snippet illustrating creating a new Spatialite database from a pre-prepared template and making a SQL query reading some Spatialite geometry fields from the database.

1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
using System.Data.SQLite;

//...etc...

string pathTemplate = @"\path\to\spatiaLiteTemplate.db";
string destDbFilename = @"\path\to\dest.db";

// Copy the template Spatialite file to the destination Spatialite file
File.Copy(pathTemplate, destDbFilename);

// Create Spatialite SQL Connection
string connectString = "Data Source=" + destDbFilename + ";Version=3;";
SQLiteConnection connection = new SQLiteConnection(connectString);

// Open the database and load in the Spatialite extension
connection.Open();
connection.LoadExtension("mod_spatialite");

// ...etc ...

// Make some Spatialite function calls
string sql = "SELECT ST_MINX(geometry), ST_MINY(geometry), ST_MAXX(geometry), ST_MAXY(geometry) FROM somefeature ";

using (SQLiteCommand command = new SQLiteCommand(sql, connection))
{
 using (SQLiteDataReader reader = command.ExecuteReader())
 {
  while (reader.Read())
  {
   double minX = reader.GetDouble(0);
   double minY = reader.GetDouble(1);
   double maxX = reader.GetDouble(2);
   double maxY = reader.GetDouble(3);
  }
 }
}

// Close and clean up the database connection
connection.Close();
connection.Dispose();
 
//...etc... 


Monday, October 3, 2016

Replaying network packets captured in a PCAP file

Some devices broadcast GPS positions over the network and saved into a PCAP file. Incidentally, PCAP stands for Packet CAPture. I got hold of some PCAP files and tried to use the Linux tcpreplay command to playback the captured network packets over the network.

The tcpreplay command input arguments require to specify the network interface and it conveniently advertises an option --listnics to list all the network interfaces on your machine. An example usage is displayed in  the screenshot below.

$ sudo tcpreplay --listnics


However, using any of the listed interfaces will result in an error when attempting to playback a pcap file, as shown below.

$ sudo tcpreplay --intf1 bluetooth0 *.pcap



After some research, I found the ip command with the link option that lists the network interfaces correctly.

$ ip link



Now plugging in the correct interface name into the tcpreplay command, my network packets are successfully replayed and broadcasted.

$ sudo tcpreplay --intf1 enp0s25 *.pcap

 

Monday, September 26, 2016

WebApp for converting local date time to GPS week and seconds of week

This online calculator can convert the local date time values into the equivalent GPS week and seconds of week. The GPS leap seconds from 1980 till 2016 are taken into account in the conversion.


To use this calculator,
  1. In the Settings pane on the right, select your local time zone, e.g. GMT+0800.
  2. Choose the local date time format, e.g. YYYY-MM-DD HH:MM:SS.
  3. Choose the output delimiter format, e.g. comma.
  4. In the Input local time field, type in one or more local date time values.
  5. Click the Convert to GPS Week Seconds button.

    The converted results are shown in the output field below.

Monday, September 19, 2016

Create a Windows Installer using CMake and CPack

CMake can be used to create a Windows installer with CPack and NSIS (Nullsoft Scriptable Install System). NSIS can be downloaded from http://nsis.sourceforge.net. This post follows and extends the tutorial from the CMake Wiki at https://cmake.org/Wiki/CMake:Component_Install_With_CPack.

I wanted to build a Windows installer for a 64 bit Visual Studio application that was build-managed using CMake. This post makes use of the example source code downloadable from https://cmake.org/Wiki/File:ComponentExampleStart.zip.

Add in CPACK macros to CMakeLists.txt
  1. Download and extract the example zip file into a folder e.g. D:\Temp\ComponentExampleStart\ as shown below. I then created a build sub-folder build underneath the D:\Temp\ComponentExampleStart\Source\ folder.
  2. Use a text editor and edit the CMakeLists.txt file to add in CPACK macros according to the Wiki. An edited example is shown below.
cmake_minimum_required(VERSION 2.6.0 FATAL_ERROR)
project(MyLib)

add_library(mylib mylib.cpp)

add_executable(mylibapp mylibapp.cpp)
target_link_libraries(mylibapp mylib)

install(
  TARGETS mylib 
  ARCHIVE
  DESTINATION lib
  COMPONENT libraries
  )
install(
  TARGETS mylibapp
  RUNTIME
  DESTINATION bin
  COMPONENT applications
  )
install(
  FILES mylib.h
  DESTINATION include
  COMPONENT headers
  )
#
# CPACK macros below here
#
set (CPACK_PACKAGE_NAME "MyLib")
set (CPACK_PACKAGE_VENDOR "CMake.org")
set (CPACK_PACKAGE_DESCRIPTION_SUMMARY "MyLib - CPack Component Installation Example")
set (CPACK_PACKAGE_VERSION "1.0.0")
set (CPACK_PACKAGE_VERSION_MAJOR "1")
set (CPACK_PACKAGE_VERSION_MINOR "0")
set (CPACK_PACKAGE_VERSION_PATCH "0")
set (CPACK_PACKAGE_INSTALL_DIRECTORY "CPack Component Example")

# Define components and their display names
set (CPACK_COMPONENTS_ALL applications libraries headers)
set (CPACK_COMPONENT_APPLICATIONS_DISPLAY_NAME "MyLib Applications")
set (CPACK_COMPONENT_LIBRARIES_DISPLAY_NAME "Libraries")
set (CPACK_COMPONENT_HEADERS_DISPLAY_NAME "C++ Headers")

# Human readable component descriptions
set (CPACK_COMPONENT_APPLICATIONS_DESCRIPTION
  "An extremely useful application that makes use of MyLib")
set (CPACK_COMPONENT_LIBRARIES_DESCRIPTION
  "Static libraries used to build programs with MyLib")
set (CPACK_COMPONENT_HEADERS_DESCRIPTION
  "C/C++ header files for use with MyLib")

# Define dependencies between components
set (CPACK_COMPONENT_HEADERS_DEPENDS libraries)

# Define groups
set(CPACK_COMPONENT_APPLICATIONS_GROUP "Runtime")
set(CPACK_COMPONENT_LIBRARIES_GROUP "Development")
set(CPACK_COMPONENT_HEADERS_GROUP "Development")

set(CPACK_COMPONENT_GROUP_DEVELOPMENT_DESCRIPTION
   "All of the tools you'll ever need to develop software")

# Define NSIS installation types
set(CPACK_ALL_INSTALL_TYPES Full Developer)
set(CPACK_COMPONENT_LIBRARIES_INSTALL_TYPES Developer Full)
set(CPACK_COMPONENT_HEADERS_INSTALL_TYPES Developer Full)
set(CPACK_COMPONENT_APPLICATIONS_INSTALL_TYPES Full)
 
# Must be after the last CPACK macros
include(CPack)

Generate the Visual Studio solution

  1. Open up a Command Prompt. Change to the build directory.

    D:> cd \temp\ComponentExampleStart\Source\build
  2. Use the cmake command to generate the build files.

    D:> cmake -G "Visual Studio 14 2015 Win64" ..

    The build files are generated.
Build the Visual Studio solution and generate the installer
  1. Using Visual Studio, open up the generated solution e.g. MyLib.sln.

  2. Since I wanted a release build, in Visual Studio I changed from Debug to Release, as shown below. Then I selected Build.

    The release binaries are generated.
  3. Next, in the Solution Explorer, select and mouse right click on the PACKAGE project.

    A pop down menu appears.
  4. Choose Build.



    CPack processing message appears.


    The NSIS installer is generated.


Monday, September 12, 2016

C++ example code for reading an Sqlite database from multiple threads

SQLite works well with threads as long as a single thread has its own database connection, or so they say in the documentation. I wanted to make a lot of database read only select calls on a table from multiple threads and I coded my C++ program according to the guidelines. But I found out some of the threads will get a "database is locked" exception after a while, and any subsequent calls to the database will raise a "library routine called out of sequence" exception. Changing the SQL connection to other types such as SQLITE_OPEN_FULLMUTEX, SQLITE_OPEN_NOMUTEX, and or permutations did not solve my problem.

Eventually I decided on a workaround to give each thread an in-memory copy of the database. The following code snippets show a simple example. The example uses the light weight C++ SQLite wrapper from https://github.com/aminroosta/sqlite_modern_cpp

The runThread function creates a copy of the original source database example.sqlite and puts it in memory with a unique connection string. Subsequently, it performs some read operations on the in-memory database.
1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
int runThread(int threadNumber) {
 try {
  cout << "[" << threadNumber << "]" << "Running" << endl;
  stringstream ss;

  // Form the connection string to the in-memory database
  ss << "file:memory" << threadNumber << "?mode=memory";
  string inMemoryDbConnectString = ss.str();
  ss.str("");

  // Open a database connection to the in-memory database
  database inMemoryDb(inMemoryDbConnectString);

  // Open a database connection to the original source database
  database sourceDb("example.sqlite");

  // Copy the source database to memory
  auto sourceConnection = sourceDb.connection();
  auto state = 
   std::unique_ptr<sqlite3_backup, decltype(&sqlite3_backup_finish)>(
    sqlite3_backup_init(inMemoryDb.connection().get(), "main", sourceConnection.get(), "main"),
    sqlite3_backup_finish
    );
  if (state) {
   int rc;
   // Each iteration of this loop copies 500 database pages from database db to the backup database.
   do {
    rc = sqlite3_backup_step(state.get(), 500);
//std::cout << "Remaining " << sqlite3_backup_remaining(state.get()) << "/" << sqlite3_backup_pagecount(state.get()) << "\n";
   } while (rc == SQLITE_OK || rc == SQLITE_BUSY || rc == SQLITE_LOCKED);

   // Do some reading from the in-memory database table
   for (int i = 0; i < 10000; i++) {

    double seconds = 452220.0 + (double)threadNumber + (double)i;
    double lower = seconds - 1;
    double upper = seconds + 1;

    ss
     << "SELECT seconds, x, y, z FROM attendance WHERE seconds >="
     << lower
     << " AND seconds <="
     << upper
     << " ORDER BY seconds ASC";

    inMemoryDb << ss.str() >> [&](
     double seconds, double x, double y, double z) {

     // do something with the returned records

    };
    ss.str("");
   }
  }
 }
 catch (sqlite_exception &e) {
  cout << "[" << threadNumber << "]" << "Error:" << e.what() << endl;
 }
 cout << "[" << threadNumber << "]" << "End" << endl;

 return 0;
}


Finally, the main function that creates multiple threads and waits for them to complete.

1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
int main(int argc, char **argv)
{
 // Create threads and put them into an array
 thread threads[3];
 for (int i = 0; i < 3; i++) {
  threads[i] = thread(runThread, i);
 }

 // Wait for the threads to complete
 for (int i = 0; i < 3; i++) {
  threads[i].join();
 }

 return 0;
}

An screenshot of the program in execution. Since each thread has a copy of the database, the database is never locked by other threads and I will never encounter the "database is locked" exception.

Note that this workaround will not work if read and write operations are required.


Related Posts with Thumbnails