3.4 Data Sources

There are many ways to enter posts into your Ushahidi deployment other than from the deployment itself. This section shows you how to configure all the possible data source types.

NB: If you’re a user on ushahidi.com, the number of data source types available to you may be limited, based on the Ushahidi plan you are subscribed to. You may review these from **[_our plans page](https://www.ushahidi.com/pricing). For open source/self hosted deployments, all data source types are available to you**_

To access the data sources configuration page,

  • On the left hand menu bar, click on Settings

  • Then, click on Data Sources.

Data sources link in the deployment's settings page.
  • You should get a full list of data sources as shown below

List of available datasources

3.4.1 Email

This section allows you to set up the platform to receive emails from users. Before getting started, make sure that you have an email account set up on Gmail, Yahoo or any other service provider. Make sure that you have IMAP/POP enabled (For more information on these two protocols, visit this website. Instructions on how to enable the IMAP/POP settings in your email can be found here

To get started with email set up,

  • Click on the drop down icon on the right as shown

  • Input the following email account settings:-

    • Incoming server type: You have two options to select from, POP and IMAP. We recommend using IMAP if possible because it’s the best way to make sure you can see all your mail at any time on all of your devices

    • Incoming server: Enter the address of the server where your email services are hosted. E.g mail.yourwebsite.com, imap.gmail.com or pop.gmail.com

    • Incoming server port: Enter the port that your email account uses for incoming emails. This is also provided by your service provider and depends on the use of SSL(Secure Sockets Layer)/Transport Layer Security(TLS) or not. As a standard rule;

      • IMAP uses port 143 , but SSL/TLS encrypted IMAP uses port 993 .

      • POP uses port 110 , but SSL/TLS encrypted POP uses port 995

    • Incoming server security: You have 3 options to choose from to enhance secure connection to your email mailbox, depending on which is supported by your email service provider.

    • Incoming user name: Enter the email address you want to use to receive emails e.g [email protected]. We recommend setting up a separate email address for this purpose, preferably one that has a lot of available space to avoid the account getting full in a short time, especially if the platform will be receiving a lot of submissions via email.

    • Incoming password: Enter the password of the email account inserted above.

Email Datasource configuration.
  • Click on Save **and this data source’s settings will be saved. Unstructured posts from email will now be pulled into the platform.

  • To enable/disable the email data source, simply click on the green toggle.

  • If you’d like to edit your email configuration, simply click on the drop down icon on the right while on the data sources list page and make your changes.

Nb: Reports coming via email will take a while before they reflect on the platform because there is a “job” that needs to be run in the database and those jobs run every 10 minutes or so.

3.4.2 Gmail

This data source is not yet available in the stable versions of Platform. You may not find it in your settings.

This section allows you to set up the platform to receive and send emails through a connected Gmail account via the API. Before getting started, make sure that you have an email account set up on Gmail.

Nb: If you're self hosting the Ushahidi platform, the Gmail Provider makes use of the server-side flow when making a request to Gmail API, and must be authorized using OAuth 2.0 Credentials.

  • Follow this guide to create the OAuth 2.0 Credentials (Client ID and Client Secret) with Gmail. Keep in mind:

    • Select "where you will be calling the API from" as Other UI.

    • Select User data as the data type to be accessed.

    • Select your application type as TVs and Limited Input devices.

    • There's a need to set up a consent screen.

To get started with Gmail set up.

  • Click on the drop down icon on the right as shown

  • Input the following email account settings:-

    • Email Address: Enter the Gmail address you will be receiving and sending emails with.

    • Fetch Email From: Pick the date you want to start fetching messages from.

    Nb: The following fields are available for only self hosted ushahidi platform, where there is a need to manually provide the Gmail API credentials.

    • Client Id: Enter the Gmail client id gotten from the OAuth credentials.

    • Client Secret: Enter the Gmail client secret gotten from the OAuth credentials.

    • Redirect URL: Enter the redirect URL from the OAuth credentials. The default value set here is urn:ietf:wg:oauth:2.0:oob and in most cases should be left this way.

  • Click on Save and this data source’s settings will be saved.

  • Next, Scroll up and click on Connect your Gmail Account and this will show a google prompt to authenticate your Gmail account.

  • Follow the prompt through and Grant permission for the platform to access your Gmail account. This will display an auth code and the end of the process.

  • Please copy the provided code, switch back to your Ushahidi application, and paste/enter it on the opened modal:

  • Click on Authorize to save the auth code.

  • Done with connecting your Gmail account, messages from Gmail will now be pulled into the platform via the API.

  • To enable/disable the email data source, simply click on the green toggle.

3.4.3 FrontlineSMS

FrontlineCloud is an online SMS management platform that lets you control and manage 2-way SMS engagement with anyone in the world.

NB: You need a FrontlineCloud Starter or Pro account to be able to configure this as a data source. To sign up, go to https://cloud.frontlinesms.com **

You also need to create a FrontlineSync connection with your mobile phone as a gateway with FrontlineSMS

To set up your Frontlinesync using your mobile phone;

  • Download FrontlineSync from the Google play store

  • You need to accept all permissions that it requests unfortunately or it won't run

  • Log in with your Frontline SMS account that you created on the FrontlineCloud platform.

Enter your FrontlineSMS account credentials
  • Then you need to select how you want the sync to work. You will be presented with a few options.

Definition for each option are referenced from the FrontlineSMS documentation

  • Send messages using this Android

    • "This option sets whether or not messages are sent from FrontlineCloud or FrontlineSMS to recipients using the Android device as an SMS gateway. If this option is not active, SMS messages will not be sent via FrontlineSync to recipients. Messages sent via FrontlineSync will appear in the native 'Sent Messages' folder within Android phones. There may be limits on the number of messages that the Android device can send over a certain period. Please note these restrictions are from the Android phone, and vary handset to handset"

  • Upload incoming messages from FrontlineSync

    • "This option sets whether or not incoming SMS to your Android device are sent to your FrontlineCloud or FrontlineSMS workspace. Any and all messages received to the device while FrontlineSync is running will automatically be uploaded to your workspace. If you disabled the app, upon reactivating FrontlineSync, you will be prompted as to whether you wish to upload all SMS received while the app was off. FrontlineSync will attempt to upload an SMS as soon as it arrives on the phone. If unsuccessful, it will retry until successful. The state of every message is maintained and visible in the Activity Log."

  • Upload missed calls from FrontlineSync [you can leave this one off]

  • It also has an option to select how often you want the synchronization to happen. By default it's 15minutes.

Select connection options
  • After selecting the options you want, click on update to finish configuring Frontlinesync.

  • Click on 'Done! start using FrontlineSync', you will be taken to a dashboard screen that shows which options are selected.

FrontlineSync connection options selected

WARNING: FROM THE POINT, FRONTLINE SYNC WILL START SYNCHRONIZING NEW MESSAGES AS THEY COME IN.

  • You can change your settings by going to the options in Frontline Sync.

    • Click the gear icon on the top-right side of the screen

    • Click "Settings"

    • You will be taken to a list, tap Options to re-configure your synchronization options.

Settings Screen

Now let's get started with FrontlineCloud setup,

  • Log into your FrontlineCloud account on https://cloud.frontlinesms.com

  • Open your deployment as well i.e name.ushahidi.io, go to datasource configuration in your setting page then click on FrontlineSMS.

Data sources → Frontline SMS → highlighted arrow button to add configuration details for SMS provider.

FrontlineSMS has announced its imminent cessation of service, with a subsequent release of the source code for its server. Presumably, users will be able to download the code and install their own servers.

As of this writing (September 2021), to our knowledge, the source code or any of the files necessary for a server installation haven't been made public yet.

If you are still enjoying service from FrontlineSMS, in their cloud instance with the address https://cloud.frontlinesms.com/, you should use that address as your server address below.

  • First, you will need to set up your server. Once you do that, copy the server URL and enter it in the 'FrontlineSMS server URL' field in your deployment.

  • Click on save.

  • Enable/disable the email data source by clicking on the green toggle that appears.

  • Click on the toggle to import your messages' response to a survey.

  • Select the survey you wish to import your messages to.

  • Select which field should be assigned to populate the data from

Enter the FrontlineSMS server URL & enable configuration.
  • Next, you will need to generate an API Key in Frontline SMS. To do so, follow the instructions below

APi web services and integrations
  • Click on API Web services and integrations.

  • In the screen that comes up, click Connect a web service

Connect a web service
  • You will be shown a list of service types. Click on Connect an external web service to your workspace.

Connect an external web service to your workspace
  • In the popup to create it, just fill in a name for the service. Example: "Ushahidi deployment integration"

Enter a name for the web service integration
  • Click Save

  • Once you save this, you will see your API key was created. You can copy it to use it in your integration with Ushahidi Platform.

Copy the generated API key
  • In the Ushahidi Platform Datasource configuration for Frontline SMS;

    • Enter the FrontlineCloud API KEY that you generated in the previous step in the Api Key field.

Enter API key
  • In the Frontline SMS website, go to Activities

Activities and create an activity in the FrontlineSMS website
  • Click Create an Activity

  • You will get a screen with a lot of activity options. The one you want is Forward to URL as highlighted below

Click on Forward to Url
  • You will get a popup to configure the URL forward activity.

Name your activity, choose a parameter, add the target URL and select HTTP method
  • Let's fill in the popup:

    • Name your activity: use a descriptive name that contains your Ushahidi deployment name + Forward SMS. Example: "Forward SMS to my deployment". This is just so that in the future you know what's done by an activity vs another.

    • Send request parameters for (You can use the one that fits your needs, by default we are using All inbound SMS in phones that are exclusive for this campaign).

      • SMS Messages that start with the specified keywords: this can be useful if you are running multiple campaigns and want to route SMS received by a specific issue. For example, you could decide that only SMS starting with "ELECTIONS2020" will be sent to this Ushahidi deployment.

      • SMS Messages that contain the specified keywords: the same as above but instead of only routing SMS that starts with "ELECTIONS2020" you would route any messages that contain that text anywhere. This can be a tricky configuration and generally, it's better to do specific keywords at the beginning instead if you are filtering.

      • All inbound SMS: all SMS will be sent to the deployment.

      ****

    • Target URL: This is the API Url to your deployment + the endpoint to connect to.

    For Ushahidi.io deployments, the rule is always the same. Given a deployment with the following url "https://**yourdeploymentname**.[ushahidi.io](http://ushahidi.io)" + the API Url will be = "https://**yourdeploymentname.api**.[ushahidi.io](http://ushahidi.io)".

  • Then you will need to add key and values that Frontline SMS will use when connecting with the Ushahidi deployment. To start;

    • Click on the Add Request parameter button which will add a 3rd key and value fields to the form.

Click on Add Request Parameter
  • In the first Key field enter from. In the Value field next to it, enter ${trigger.sourceNumber}

    • This will be saved as the sender for the SMS

  • In the second Key field enter message. In the Value field next to it, enter ${trigger.text}. This matches the content of the SMS sent.

  • In the third Key field enter secret. In the value field next to it, enter a phrase or long password. This will be needed in the Ushahidi Platform as well. Example "occupant-gash-gasket-roil" (don't use this! it's just an example). The secret is supposed to be secret, and it validates the communication between Frontline SMS and the Ushahidi API. Copy the secret into the Ushahidi Platform Frontline SMS configuration as well. Remember: it's case sensitive.

Fill in the request parameters
  • Click on the Save button in Frontline SMS to save the activity.

  • Enter a secret code in the SECRET field. This code will be used by FrontlineCloud to connect to the Ushahidi deployment. You can enter any value you like here. It should ideally contain numbers and letters and be ~20 characters long.

Enter your secret value
  • Click save when done.

  • Finally, test that the system is setup correctly by sending a test sms to your number. The message should be forward to the Ushahidi deployment and appear in your list of posts.

3.4.4 Nexmo

Nexmo is a cloud-based SMS API that lets you send and receive a high volume of messages to mobile phones in any country at wholesale rates.

NB: You need a nexmo account to be able to configure this as a data source. To sign up, go to (https://dashboard.nexmo.com/sign-up)

To get started with Nexmo set up,

  • Log into your Nexmo Dashboard https://dashboard.nexmo.com

  • If you haven’t already, you’ll need to buy a number that you will use to receive SMS messages from.

    • Click on Numbers on the top menu bar on your nexmo dashboard

Click on numbers
    • Click on Buy Numbers

Buy Number
    • Set the desired criteria of the phone number you’re looking to use

Set desired criteria
      • Select the country in which the SMS Number will likely be operating in

      • Select the features of this phone number(SMS only, Voice only or SMS & Voice)

      • Select the type of phone number it will be (Mobile, Landline, Toll free)

    • Click on search. A list of available numbers based on the criteria set above will appear.

Search for an available number to purchase
  • Click Buy on the number you’d like to use.

  • Once you have a phone number, note it down as you’ll need it to configure your data source later on.

  • You’ll need to grab your API credentials from your nexmo settings page.

Click on your Profile
Click on settings
  • Pick your API KEY and API SECRET from the API Settings section.

API settings
  • Go back to your Data source settings page on your deployment

  • Click on the drop down icon on the right to get to your Nexmo configuration page

Data sources -> arrow button in the Nexmo SMS provider.
  • Enter the following details, which you got earlier from your Nexmo Dashboard

    • From: Enter the phone number you will use to receive SMS messages from your nexmo account

    • API KEY: Enter the API key retrieved from your nexmo settings page.

    • API SECRET: Enter the API secret retrieved from your nexmo settings page.

  • Click on Save **and this data source’s settings will be saved. Unstructured posts from SMS will now be pulled into the platform from Nexmo.

Highlighted: Nexmo configuration in the Ushahidi datasource configuration page.
  • To enable/disable the nexmo data source, simply click on the green toggle.

Toggle enabled to accept survey submissions from Nexmo.
  • If you’d like to edit your nexmo configuration, simply click on the drop down icon on the right while on the data sources list page and make your changes.

3.4.5 SMSSync

We are having some issues with the play store at the moment, you can download the SMS Sync app at https://github.com/ushahidi/SMSSync/releases/download/v3.1.1/smssync-withAnalyticsRelease-v3.1.1-RELEASE.apk

SMSsync is a simple, yet powerful SMS to HTTP sync utility that turns any Android phone into a local SMS gateway by sending incoming messages (SMS) to a configured URL (web service).

Please note that SMSsync works on any SMS-enabled device running Android 2.1 and above.

  • Once you have downloaded it, click on install to save it on your phone.

Installing SMSsync App on your phone
  • Open up the SMSSync Application on your android device, click skip or swipe left, and click done to continue.

SMSsync interface
  • On the incoming messages screen, click on the arrow at the bottom right corner to enable SMSsync.

Enable SMSsync as your sms platform service
  • Once you have enabled SMSsync, you will be redirected to the integrations screen. Start the SMSSync Service to start fetching messages in your app.

  • Click on the custom web service to configure SMSsync with your deployment.

Integration screen
  • Manage custom web services screen will open, you’ll note that you can manage multiple Sync URLs on the app.

  • To add a new Sync URL, tap on the Add icon on the actionbar. An input dialog should open.

Configuring SMSsync datasource on your Ushahidi Platform

  • Open your deployment i.e name.ushahidi.io, go to datasource configuration in your setting page.

  • Click on the drop down icon on the right as shown

Settings - > data sources: arrow button highlighted to access the SMS Sync data source configuration
  • Follow the instructions given to you below.

Android App settings
  • You will need the Sync URL to configure SMSSync.

  • Set a SMSSync secret key for security purposes.

  • Click on Save.

  • To enable/disable this datasource , simply click on the green toggle to accept survey submissions.

Enabling SMSsync as a datasource on your Ushahidi platform
  • Click on the toggle to import your messages' response to a survey.

  • Select the survey you wish to import your messages to.

  • Select which field should be assigned to populate the data from SMSsync.

SMSsync setup for Ushahidi Platform SMS connectivity

  • Go back to the input dialog on your SMSsync App and add the following details;

Custom web service configuration settings
  • Title for the custom service: This can be the name of your deployment or any title you see fit

  • Custom web service Url: Enter the Sync URL in step 2 from your deployment as shown in the 'Android settings' image above i.e "https://yourdeploymentname.api.ushahidi.io/sms/smssync".

  • Secret key: Make sure you enter the exact key as the one you entered in your deployment android setting as shown in the 'Android settings' image above.

  • Once you enter your secret key, you are done with the configuration, do not change the rest of the settings.

  • Click ADD or test integration to save the new entry.

Nb:The secret key should be presented as string of any characters without spaces. Enter a comma separated value for the keyword(s). These keywords will be used by SMSsync to filter incoming SMS and pending messages to the Sync URL you are adding. As of v2.0.2. You can now add Regular Expression code for filtering. This means, it can either be CSV or RegExp. It cannot be both.

Note: Version 2.5 or higher supports basic auth credentials in the URL, e.g. https://username:[email protected]/api-v1/add-record/.

  • You will now need to start the SMSSync Service to start forwarding messages to the platform. To start the SMSSync service

    • Make sure that you have added and enabled(checked) the Sync URL you added above.

    • On the integration screen, tap on the Start SMSsync service to start the service if you had not enabled it.

You should be all set to work with SMSSync and Ushahidi now. Unstructured posts via SMS will now be pulled into the platform.

If you’d like to edit your SMSSync configuration, simply click on the drop down icon on the right while on the data sources list page and make your changes.

For more details on how to manage messages within SMSSync, see configuration instructions on the SMSSync Website

3.4.6 Twilio

Twilio allows you to programmatically make and receive phone calls and send and receive text messages using its web service APIs.

NB: You need a Twilio account to be able to configure this as a data source. To sign up, go to **[_https://www.twilio.com/try-twilio**_](https://www.twilio.com/try-twilio)

To get started with Twilio set up,

  • Log into your twilio account.

  • You’ll need to buy a number to use. Click on the hashtag icon

Buy a number
  • Click on the red "Plus" icon.

Click on the red (+) icon
  • Select the desired criteria for your phone number In the number selection/purchase screen

    • Select the country where you need a phone number. This is the number that people will use when sending a report through SMS.

    • Location/Number: You can search numbers by digits/phrases or by location. This is optional.

    • Capabilities (Voice, SMS, Fax, MMS): Make sure you tick the "SMS" checkbox. We want to make sure Twilio only shows us numbers with SMS capabilities. If it can't be ticked/clicked, then it means that Twilio doesn't have any numbers with SMS capabilities in that country.

Setting up a Twilio number. Capabilities selected: SMS. Select your Country, and a number. Then hit the search button.
  • Click on **_Search**_. A list of available numbers with their rates based on the criteria set above will appear

  • Click Buy on the number you’d like to use.

List of Twilio numbers. Click Buy to get one.
  • You will be shown a screen like this one below where they confirm all capabilities, the number, and other details.

Click the buy button purchase the number
  • Click the buy button at the bottom right hand of the screen to purchase your number.

  • Once you have a phone number, note it down as you’ll need it to configure your data source later on.

Configuring Ushahidi Platform to use your Twilio phone number

First, we need to get the following information from your Twilio account:

  • Phone Number: the number you just purchased. If you forgot it, you click the "Hashtag" icon in Twilio again to see your available numbers.

  • Account SID

    • You will see a panel like the one below with details about your project. Get the Account SID from it (we've highlighted it in the screenshot as a reference)

  • Auth Token

    • In the same panel as the account SID, you will see a "view" link next to "Auth token" as shown on the diagram above.

    • Click it, and it will display your account's auth token.

Next, with the Account SID, Auth Token and Phone number you just obtained:

  • Go to your Datasources settings page on your Ushahidi deployment.

  • Click on the drop down icon on the right as shown

Twilio data source configuration.
  • You will see a list of fields like this one

Twilio datasource configuration in Ushahidi Platform.
  • Enter the following details, which you got earlier from your Twilio Account

    • From: Enter the phone number you got from Twilio.

    • ACCOUNT SID: Enter the account SID of your Twilio account

    • AUTH TOKEN: Enter the Auth Token retrieved from your Twilio account Dashboard page

    • SMS Auto Response: This will be sent automatically when a user sends a message to your Twilio number. You could use it to provide reference information, like thanking the user or just confirm receipt.

    • Click on Save **and this data source’s settings will be saved. Unstructured posts from SMS will now be pulled into the platform from Twilio.

  • To enable/disable the Twilio data source and choose which survey to import to, simply click on the green toggles.

Enable survey submissions from Twilio.
  • At this point, your Twilio setup in Ushahidi Platform is ready to use but we need to do some follow up tasks in Twilio to connect both of them.

Twilio setup for Ushahidi Platform SMS connectivity

Go to https://www.twilio.com/console/phone-numbers/runtime/twiml-apps where you can create Twiml apps. This is where we can make Twilio aware of the existence of your Ushahidi deployment and help them connect with each other.

  • Click on the red "+" button to create a new app. You will see a screen like the one below.

Create TwiML App

Fill in these details;

Configure your phone number to send messages to Ushahidi platform

Configure with "TwiML App" and Enter the TWIML App friendly name
  • In the configure with field: Select TwiML App

  • in the TWIML APP field: Enter the Friendly name you created above for your deployment. This will ensure that when you get an SMS, Twilio sends a request to Ushahidi platform to save the message.

Trying this out

Checking that Twilio gets the message

  • Send an SMS to the phone number you purchased and configured (ie from your own phone).

  • You can validate that messages are getting to Twilio by going to the Programmable SMS logs https://www.twilio.com/console/sms/logs

  • You will see the messages your Twilio number has received and sent in the list

  • The list will look like this. You can click on the red link under "Date" to see the details of each message.

SMS Logs
  • Example of a single message below.

Details of a single Message

Checking that Ushahidi Platform gets the message

  • Go to the Ushahidi Platform

  • Go to the data view

  • You should see the messages you sent in the data view list

3.4.7 Twitter

This section allows you to configure Twitter as a data source, and subsequently pull unstructured posts from specific Twitter hashtags.

For you to be able to pull tweets based on hashtags, you will need to set up your Ushahidi deployment as an application on Twitter. To get started,

  • Click on the drop down icon on the right as shown

Settings -> Datasources: arrow button to open the Twitter datasource configuration
Twitter website: create a new twitter application.
Get your API Keys
  • On the module that will pop up, click on Apply to sign up for a developer account.

Apply for a twitter developer account
  • To get access to the Twitter API, you have to select the reason why you want it. You will be redirected to a page with options, in our case select Doing something else.

Pick the reason you need the twitter developer tools for
  • Click Next after checking that option.

  • Make changes or finish filling in your individual developer account profile details and click Next.

Enter your details
  • On the application page that will appear, fill in the details of how you intend to use the Twitter API key and click Next once done.

Enter your intentions
  • Review your information, if you would like to change anything, click Back but if you are done and do not want to make any changes, click it looks good.

    Review the developer's Agreement and accept the terms if you agree to the policy.

Terms of Agreement
  • Tick the checkbox to agree and conclude, click submit and submit the application.

Submit Application

A confirmation will be sent to your email, go to your email and confirm it. Once you confirm, you will be redirected to the Twitter developer platform.

  • Select your App name.

You can enter the name of your deployment to make it more memorable.

Enter your App/Deployment/Project name
  • Click on Get keys.

  • Test application configuration on Twitter.

  • Save them to access your developer's portal

Test application configuration on Twitter.

The overview page on Projects and Apps
  • You’ll get redirected to a page where you can grab details needed to configure your Ushahidi deployment i.e CONSUMER KEY, CONSUMER SECRET, ACCESS TOKEN, ACCESS TOKEN SECRET.

Grab your consumer access token & Secret and API key & Secret
  • Go back to your Twitter configuration page on your deployment and fill in all the details from your Twitter app management page.

Configuration page for Twitter in Ushahidi Platform
  • On your Twitter developer account panel, click on view keys next to API key and Secret to display them, copy and paste them on your deployment Api and secret fields.

  • On the same panel, you will see access token and secret. You’ll have to generate an ACCESS TOKEN and ACCESS TOKEN SECRET by clicking on Generate my access token and token secret.

  • Copy and paste them on on your deployment ACCESS TOKEN and ACCESS TOKEN SECRET fields.

NB: if you save them without copy and pasting, you will be regenerating new ones.

  • Add the hashtags you want to pull tweets from in the Twitter Search Terms section. You can choose more than one hashtag, separated by a comma. It is recommended that short and clear hashtags be chosen.

  • Click on Save and this data source’s settings will be saved. Unstructured posts from Twitter will now get pulled into the platform.

Configure Twitter in the Ushahidi Platform

  • To enable/disable the Twitter data source, simply click on the green toggle.

  • If you’d like to edit your twitter configuration, click on the drop down icon on the right while on the datasource list page and make your changes.

Enable the Twitter toggle in the datasource panel to get tweets into platform.

Enable the twitter toggle in the datasource to get tweets into platform

It will only take a few minutes and the tweets will appear on your deployment.