3.4 Data Sources

There are many ways to enter posts into your Ushahidi deployment other than from the deployment itself. This section shows you how to configure all the possible data source types.

NB: If you’re a user on ushahidi.com, the number of data source types available to you may be limited, based on the Ushahidi plan you are subscribed to. You may review these from our plans page. For open source/self hosted deployments, all data source types are available to you

To access the data sources configuration page,

  • On the left hand menu bar, click on Settings

  • Then, click on Data Sources.

Data sources link in the deployment's settings page.
  • You should get a full list of data sources as shown below

List of available datasources

3.4.1 Email

This section allows you to set up the platform to receive emails from user. Before getting started, make sure that you have an email account set up on Gmail, Yahoo or any other service provider. Make sure that you have IMAP/POP enabled (For more information on these two protocols, visit this website. Instructions on how to enable the IMAP/POP settings in your email can be found here

To get started with email set up,

  • Click on the drop down icon on the right as shown

  • Input the following email account settings:-

    • Incoming server type: You have two options to select from, POP and IMAP. We recommend using IMAP if possible because it’s the best way to make sure you can see all your mail at any time on all of your devices

    • Incoming server: Enter the address of the server where your email services are hosted. E.g mail.yourwebsite.com, imap.gmail.com or pop.gmail.com

    • Incoming server port: Enter the port that your email account uses for incoming emails. This is also provided by your service provider and depends on the use of SSL(Secure Sockets Layer)/Transport Layer Security(TLS) or not. As a standard rule;

      • IMAP uses port 143 , but SSL/TLS encrypted IMAP uses port 993 .

      • POP uses port 110 , but SSL/TLS encrypted POP uses port 995

    • Incoming server security: You have 3 options to choose from to enhance secure connection to your email mailbox, depending on which is supported by your email service provider.

    • Incoming user name: Enter the email address you want to use to receive emails e.g sample@youremail.com. We recommend setting up a separate email address for this purpose, preferably one that has lot of available space to avoid the account getting full in a short time, especially if the platform will be receiving a lot of submission via email.

    • Incoming password: Enter the password of the email account inserted above.

Email Datasource configuration.
  • Click on Save and this data source’s settings will be saved. Unstructured posts from email will now be pulled into the platform.

  • To enable/disable the email data source, simply click on the green toggle.

  • If you’d like to edit your email configuration, simply click on the drop down icon on the right while on the data sources list page and make your changes.

Nb: Reports coming via email will take a while before they reflect on the platform because there is a “job” that needs to be run in the database and those jobs run every 10 minutes or so.

3.4.2 FrontlineSMS

FrontlineCloud is an online SMS management platform that lets you control and manage 2-way SMS engagement with anyone in the world.

NB: You need a FrontlineCloud Starter or Pro account to be able to configure this as a data source. To sign up, go to https://cloud.frontlinesms.com.

To get started with FrontlineCloud set up,

  • Log into your FrontlineCloud account.

  • You’ll need to setup your network connection

Frontline SMS Home page
  • Click on Connect to a Mobile Network

Connect to a mobile network button highlighted in "Configure your connections" page of Frontline SMS
  • FrontlineSMS gives you several different mobile network options to end and receive SMS. Choose the most appropriate for you from the list given below.

Selecting one of the many mobile network options from Frontline SMS.
  • Once you have setup your mobile network, click on Activities

Activities button highlighted in the Frontline SMS menu.
  • Next you will need to setup An Activity to allow sending SMS from Ushahidi via Frontline

  • Setup the FrontlineSMS data source on your Ushahidi platform

    • Click on the drop down icon on the right as shown

Data sources -> Frontline SMS -> highlighted arrow button to add configuration details for SMS provider.
  • Enter the FrontlineCloud API KEY that you generated in the previous step in the Api Key field.

  • Enter a secret code in the SECRET field. This code will be used by FrontlineCloud to connect to the Ushahidi deployment. You can enter any value you like here. It should ideally contain numbers and letters and be ~20 characters long.

Highlighted: API key and secret key inputs.
  • Copy the Secret you just created (You will need it for the final FrontlineCloud configuration), then click on Save

  • Return to your FrontLineCloud account

  • Follow this guide:

  • For Step 2 part 6, you will need to define the following key, value pairs shown in the image below:

    • key:message value:${trigger.text}

    • key:from value:${trigger.sourceNumber}

    • key:secret value:<secret value>, where the secret value is the one you created on the Ushahidi datasource step. Replace <secret value> with the secret you copied earlier

Frontline SMS website: setting up the Ushahidi web service.
  • Finally, test that the system is setup correctly by sending a test sms to your number. The message should be forward to the Ushahidi deployment and appear in your list of posts.

3.4.3 Nexmo

Nexmo is a cloud-based SMS API that lets you send and receive a high volume of messages to mobile phones in any country at wholesale rates.

NB: You need a nexmo account to be able to configure this as a data source. To sign up, go to https://dashboard.nexmo.com/sign-up

To get started with Nexmo set up,

  • Log into your Nexmo Dashboard https://dashboard.nexmo.com

  • If you haven’t already, you’ll need buy a number that you will use to receive SMS messages from.

    • Click on Numbers on the top menu bar on your nexmo dashboard

    • Click on Buy Numbers

    • Set the desired criteria of the phone number you’re looking to use

      • Select the country in which the SMS Number will likely be operating in

      • Select the features of this phone number(SMS only, Voice only or SMS & Voice)

      • Select the type of phone number it will be (Mobile, Landline, Toll free)

    • Click on Search. A list of available numbers based on the criteria set above will appear.

    • Click Buy on the number you’d like to use.

  • Once you have a phone number, note it down as you’ll need it to configure your data source later on.

  • You’ll need to grab your API credentials from your nexmo settings page.

  • Pick your API KEY and API SECRET from the API Settings section.

  • Go back to your Data source settings page on your deployment

  • Click on the drop down icon on the right to get to your Nexmo configuration page

Data sources -> arrow button in the Nexmo SMS provider.
  • Enter the following details, which you got earlier from your Nexmo Dashboard

    • From: Enter the phone number you will use to receive SMS messages from your nexmo account

    • Secret: Enter a secret value for security purposes.

    • API KEY: Enter the API key retrieved from your nexmo settings page.

    • API SECRET: Enter the API secret retrieved from your nexmo settings page.

  • Click on Save and this data source’s settings will be saved. Unstructured posts from SMS will now be pulled into the platform from Nexmo.

Highlighted: Nexmo configuration in the Ushahidi datasource configuration page.
  • To enable/disable the nexmo data source, simply click on the green toggle.

Toggle enabled to accept survey submissions from Nexmo.
  • If you’d like to edit your nexmo configuration, simply click on the drop down icon on the right while on the data sources list page and make your changes.

3.4.4 SMSSync

We are having some issues with the play store at the moment, you can download the SMS Sync app at https://github.com/ushahidi/SMSSync/releases/download/v3.1.1/smssync-withAnalyticsRelease-v3.1.1-RELEASE.apk

SMSsync is a simple, yet powerful SMS to HTTP sync utility that turns any Android phone into a local SMS gateway by sending incoming messages (SMS) to a configured URL (web service).

To get started with SMSSync set up,

  • Click on the drop down icon on the right as shown

Settings - > data sources: arrow button highlighted to access the SMS Sync data source configuration
  • Follow the instructions given to you below.

Download the SMS Sync app. Turn on SMS Sync and use your deploymentName.api.ushahidi.io/sms/smssync to configure. Use the same Secret Key
  • Download the application from the Android Market by scanning the QR Code presented to you on the settings page or simply search for it in the android market.

We are having some issues with the play store version at the moment, you can download the SMS Sync app from our repository at https://github.com/ushahidi/SMSSync/releases/download/v3.1.1/smssync-withAnalyticsRelease-v3.1.1-RELEASE.apk

Please note that SMSsync works on any SMS-enabled device running Android 2.1 and above.

    • Retrieve the Sync URL, which you’ll need to configure SMSSync with under Step 2: ANDROID APP SETTINGS

    • You can also set an SMSSync secret key for security purposes

    • Click on Save

  • Open up the SMSSync Application on your android device. You’ll note that you can manage multiple Sync URLs on the app.

  • To add a new Sync URL

    • Tap on the Sync URL from the navigation drawer.

    • Tap on the Add icon icon on the actionbar. An input dialog should open.

    • Enter a title for the Sync URL.

    • Enter a secret key(If you set one above). Make sure you enter the exact key here.

    • The secret key should be presented as string of any characters without spaces.

    • Enter a comma separated value for the keyword(s). These keywords will be used by SMSsync to filter incoming SMS and pending messages to the Sync URL you are adding. As of v2.0.2. You can now add Regular Expression code for filtering. This means, it can either be CSV or RegExp. It cannot be both.

    • Enter the URL for your web service. Don't forget to start with the HTTP or HTTPS protocol. e.g. https://example.com/api-v1/add-record/

    • Tap OK to save the new entry.

Note: Version 2.5 or higher supports basic auth credentials in the URL, e.g. https://username:pass@example.com/api-v1/add-record/.

  • You will now need to start the SMSSync Service to start forwarding messages to the platform. To start the SMSSync service

    • Make sure that you have added and enabled(checked) the Sync URL you added above.

    • On the SYNC URL screen, tap on the Start SMSsync service to start the service. You only do this if the service is disabled.

  • You should be all set to work with SMSSync and Ushahidi now. Unstructured posts via SMS will now be pulled into the platform.

  • To enable/disable the SMSSync data source, simply click on the green toggle.

  • If you’d like to edit your SMSSync configuration, simply click on the drop down icon on the right while on the data sources list page and make your changes.

For more details on how to manage messages within SMSSync, see configuration instructions on the SMSSync Website

3.4.5 Twilio

Twilio allows you to programmatically make and receive phone calls and send and receive text messages using its web service APIs.

NB: You need a Twilio account to be able to configure this as a data source. To sign up, go to https://www.twilio.com/try-twilio

To get started with Twilio set up,

  • Log into your twilio account.

  • You’ll need to buy a number to use. Click on PHONE NUMBERS.

  • Click on Buy a number

  • Select the desired criteria for your phone number

    • Country

    • Location/Number

    • Capabilities (Voice, SMS, MMS)

  • Click on Search. A list of available numbers based on the criteria set above will appear

Setting up a Twilio number. Capabilities selected: SMS. Select your Country, and a number. Then hit the search button.
  • Click Buy on the number you’d like to use.

List of Twilio numbers. Click Buy to get one.
  • Once you have a phone number, note it down as you’ll need it to configure your data source later on.

  • You’ll need to grab your API credentials from your Twilio Account settings page.

Click on the Account menu button to view and edit your Twilio account settings.
  • Pick your ACCOUNT SID and AUTH TOKEN from the API Credentials section.

API Credentials.
  • Go back to your Data source settings page on your deployment

  • Click on the drop down icon on the right as shown

Twilio data source configuration.
  • Enter the following details, which you got earlier from your Twilio Account

    • From: Enter the phone number you will use to receive SMS messages from your twilio account

    • ACCOUNT SID: Enter the unique ID of your twilio account

    • AUTH TOKEN: Enter the Auth Token retrieved from your twilio settings page.

    • SMS Auto Response: This will likely be the message sent back to users who send you SMS Messages.

  • Click on Save and this data source’s settings will be saved. Unstructured posts from SMS will now be pulled into the platform from Twilio.

Twilio datasource configuration in Ushahidi Platform.
  • To enable/disable the Twilio data source, simply click on the green toggle.

Enable survey submissions from Twilio.
  • If you’d like to edit your Twilio configuration, simply click on the drop down icon on the right while on the data sources list page and make your changes.

3.4.6 Twitter

This section allows you to configure twitter as a data source, and subsequently pull unstructured posts from specific twitter hashtags.

For you to be able to pull tweets based on hashtags, you will need to set up your ushahidi deployment as an application on twitter. To get started,

  • Click on the drop down icon on the right as shown

  • Click on Create a new twitter application. This will redirect you to https://apps.twitter.com

  • Sign into https://apps.twitter.com using your twitter username and password

  • Click on “Create New App”

Settings -> Datasources: arrow button to open the Twitter datasource configuration
Twitter website: create a new twitter application.
  • Fill in the application details

    • Name – this can be your deployment/site name e.g Uchaguzi

    • Description – this is your deployment/site description – what your deployment does

    • Website – this is your deployment url/link i.e http://yourdeployment

    • Callback url – Leave this blank.

    • Agree to the terms and conditions then click on Create your twitter application

  • Once your application has been successfully created, you should now be able to access your access keys and tokens. To do so, click on Keys and Access Tokens or Manage Keys and Access

Test application configuration on Twitter.
  • You’ll get redirected to a page where you can grab details needed to configure your Ushahidi deployment i.e CONSUMER KEY, CONSUMER SECRET, ACCESS TOKEN, ACCESS TOKEN SECRET.

  • You’ll have to generate an ACCESS TOKEN and ACCESS TOKEN SECRET by clicking on Generate my access token and token secret. This may take a couple of minutes, and your page will refresh with all the details you require.

  • Go back to your twitter configuration page on your deployment and fill in all the details from your twitter app management page.

  • Add the hashtags you want to pull tweets from in the “Twitter Search Terms” section. You can choose more than one hashtag, separated by a comma. It is recommended that short and clear hashtags be chosen.

  • Click on Save and this data source’s settings will be saved. Unstructured posts from twitter will now get pulled into the platform.

Configuration page for Twitter in Ushahidi Platform
  • To enable/disable the twitter data source, simply click on the green toggle.

  • If you’d like to edit your twitter configuration, simply click on the drop down icon on the right while on the data sources list page and make your changes.

Enable the twitter toggle in the datasource to get tweets into platform.