Tips and Advice

Read the following guides to familiarize yourself with ODE's Docker and Kafka modules.

• Docker README
• Kafka README

• Data Flow Diagrams README

• Docker builds may fail if you are on a corporate network due to DNS resolution errors.

• Additionally git commands may fail for similar reasons, you can fix this by running export GITSSLNO_VERIFY=1.
• Windows users may find more information on installing and using Docker here.
• A compatibility guide containing recommendations for suitable versions of submodules for each main module version can be found here.

If you wish to change the application properties, such as change the location of the upload service via ode.uploadLocation.* properties or set the ode.kafkaBrokers to something other than the $DOCKERHOSTIP:9092, or wish to change the log file upload folder, etc. instead of setting the environment variables, modify jpo-ode-svcs\src\main\resources\application.yaml file as desired. To adjust the settings in your unit/integration tests, modify the jpo-ode-svcs\src\test\resources\application.yaml file.

ODE configuration can be customized for every deployment environment using environment variables. These variables can either be set locally or using the sample.env file. Instructions for how to use this file can be found here.

Important!

You must rename sample.env to .env for Docker to automatically read the file. This file will contain AWS access keys and other private information. Do not push this file to source control.

Back to top

3. Installation

The following instructions describe the minimal procedure to fetch, build, and run the main ODE application. If you want to use the privacy protection module and/or S3 depositors, see the User Guide for more detailed information. Additionally, different build processes are covered at the bottom of this section.

Step 0 - For Windows Users Only

If running on Windows, please make sure that your global git config is set up to not convert end-of-line characters during checkout.

Disable git core.autocrlf (One Time Only)

git config --global core.autocrlf false

Step 1 - Download the Source Code

The ODE software system consists of the following modules hosted in separate Github repositories:

| Name | Visibility | Description | |-------------------------------------------------------------------------|------------|--------------------------------------------------------------------------------------------------------| | jpo-ode | public | Contains the public components of the application code. | | jpo-cvdp | public | Privacy Protection Module | | asn1_codec | public | ASN.1 Encoder/Decoder module | | jpo-security-svcs | public | Provides cryptographic services. | | jpo-sdw-depositor | public | SDW depositor service. Optional, can be enabled by using the COMPOSEPROFILES environmental variable | | jpo-asn-pojos | public | ASN.1 POJOs for J2735 messages. |

You may download the stable, default branch for ALL of these dependencies by using the following recursive git clone command:

git clone --recurse-submodules https://github.com/usdot-jpo-ode/jpo-ode.git

If you have already cloned the repository, you can use the following command to download the stable, default branch for all dependencies by using the following command:

git submodule update --init --recursive

Once you have these repositories obtained, you are ready to build and deploy the application.

Downloading the source code from a non-default branch

# Run the following commands to reset existing branch
git reset --hard
git submodule foreach --recursive git reset --hard
Pull from the non-default branch
git checkout <branch_name>
git pull origin <branch_name>
The next command wipes out all of the submodules and re-initializes them.
git submodule deinit -f . && git submodule update --recursive --init

# Remove references to the git submodule
git submodule deinit -f jpo-s3-deposit
rm -rf .git/modules/jpo-s3-deposit
rm -rf jpo-s3-deposit
git rm --cached jpo-s3-deposit
Run the following commands to reset existing branch
git reset --hard
git submodule foreach --recursive git reset --hard
The next command wipes out all of the submodules and re-initializes them.
git submodule deinit -f . && git submodule update --recursive --init

Step 2 - Build and run the application

Notes:

• Docker builds may fail if you are on a corporate network due to DNS resolution errors.

• In order for Docker to automatically read the environment variable files, you must:

Make:

Navigate to the root directory of the jpo-ode project and run the following command:

# View available options:
$ make
Make target options:
make start to run the ODE
make build to build the ODE
make stop to stop the ODE
make delete to stop the ODE and remove the volumes
make restart to restart the ODE
make rebuild to stop, delete, and then rebuild the containers
make clean-build to rebuild the containers without using the cache

Possible error messages if .env files are not created:

# jpo-ode .env is missing
$ make start
Makefile:11: * "ERROR: jpo-ode Environment file .env not found in ".  Stop.
jpo-utils .env is missing
$ make start
Makefile:14: * "ERROR: jpo-utils Environment file .env not found in ".  Stop.

Docker Compose:

Navigate to the root directory of the jpo-ode project and run:

make start

OR

docker compose up --build -d
docker compose ps

To bring down the services and remove the running containers run:

make stop

OR

docker compose down

For a fresh restart, run:

make rebuild

OR

docker compose down
docker compose up --build -d
docker compose ps

To completely rebuild from scratch, run:

make rebuild

OR

docker compose down
docker compose rm -v
docker compose up --build -d
docker compose ps

Check the deployment by running docker compose ps. You can start and stop containers using docker compose start and docker compose stop commands. If using the multi-broker docker compose file, you can change the scaling by running docker compose scale <container>=n where container is the container you would like to scale and n is the number of instances. For example, docker compose scale kafka=3.

To configure what services are started, use the COMPOSEPROFILE environmental variable and set a comma separated string of profiles you want to start up. This project also supports all COMPOSEPROFILE values implemented in the jpo-utils docker-compose.yml. The following are the available profiles that the ODE is currently configured to use along with the services they will enable:

• Profile name: ode_base

• Profile name: ode_full

Profiles are also available for each service name to individually specify a service to enable.

asn1_codec Module (ASN.1 Encoder and Decoder)

ODE requires the deployment of asn1codec module. ODE's docker-compose.yml file is set up to build and deploy the module in a Docker container. If you wish to run asn1codec module outside Docker (i.e. directly on the host machine), please refer to the documentation of asn1_codec module.

The only requirement for deploying asn1codec module on Docker is the setup of two environment variables DOCKERHOSTIP and DOCKERSHARED_VOLUME.

PPM Module (Geofencing and Filtering)

To run the ODE with PPM module, you must install and start the PPM service. PPM service communicates with other services through Kafka Topics. PPM will read from the specified "Raw BSM" topic and publish the result to the specified "Filtered Bsm" topic. These topic names are specified by the following ODE and PPM properties:

- ODE properties for communications with PPM (set in application.yaml) - ode.kafka.topics.json.bsm (default = topic.OdeBsmJson) - ode.kafka.topics.json.bsm-filtered (default = topic.FilteredOdeBsmJson) - PPM properties for communications with ODE (set in yourconfig.properties) - privacy.topic.consumer (default = j2735BsmRawJson) - privacy.topic.producer (default = j2735BsmFilteredJson)

Follow the instructions here to install and build the PPM service.

After a successful build, use the following commands to configure and run the PPM

cd $BASEPPMDIR/jpo-cvdp/build
$ ./bsmjson_privacy -c ../config/ppm.properties

Confluent Cloud Integration

Rather than using a local kafka instance, the ODE can utilize an instance of kafka hosted by Confluent Cloud via SASL.

Environment variables

Purpose & Usage

• The DOCKERHOSTIP environment variable is used to communicate with the bootstrap server that the instance of Kafka is running on.

• The KAFKA_TYPE environment variable specifies what type of kafka connection will be attempted and is used to check if Confluent should be utilized. If this environment variable is not set, the ODE will default to normal behavior.

• The CONFLUENTKEY and CONFLUENTSECRET environment variables are used to authenticate with the bootstrap server. If the KAFKA_TYPE environment variable is not set, then these are not required.

Values

In order to utilize Confluent Cloud:

• DOCKERHOSTIP must be set to the bootstrap server address (excluding the port)

• KAFKA_TYPE must be set to "CONFLUENT"

• CONFLUENT_KEY must be set to the API key being utilized for CC

• CONFLUENT_SECRET must be set to the API secret being utilized for CC

Note

Port Mapped Receiver Configuration

To use port-mapped ingest, add an entry similar to the below to the application.yaml file. The entry below creates a BSM receiver which listens on port 1234. Any BSM data received on port 1234 will be marked as having been received from the IP 1.2.3.4.

port-mapped-ingest:
  sources:
    - intersectipon-name: "Name"
      intersection-id: 1234
      origin-ip: 1.2.3.4
      port: 1234
      type: BSM

Please note, ports added here are not automatically opened in the host docker container. Any port added here should separately be added to the running docker container.

This has only been tested with Confluent Cloud but technically all SASL authenticated Kafka brokers can be reached using this method.

Back to top

MongoDB Integration

Description and Configuration

To sink streamed kafka topic data to a MongoDB database, a kafka connect and MongoDB instance can be deployed for the ODE. To deploy the kafka connect and MongoDB instance, add the following profiles to your COMPOSEPROFILES env variable: odebase,kafkaconnectstandalone,kafka_setup. This will result in the following topics being synced to MongoDB:

• OdeBsmJson
• OdeMapJson
• OdeSpatJson
• OdeTimJson
• OdePsmJson

For further documentation on configuring the MongoDB Kafka Connect image refer to this readme.

Note

Kafka connect is being used for MongoDB in this implementation but it can interact with many types of databases, here is further documentation for kafka connect

Back to top

4. File Manifest

This section outlines the software technology stacks of the ODE.

Containerization and Management

• Docker
• Docker-Compose

Messaging

• Kafka

Code Quality

• SonarCloud

Continuous Integration

• GitHub Workflows

ODE Code

• Java 21
• Maven
• Spring Boot
• Logback
• SNMP4J
• JUnit
• JMockit - deprecated in favor of Mockito
• Mockito
• Stomp Websocket

Web UI

• MaterializeCSS
• jQuery
• Stomp Websocket
• SockJS

5. Development Setup

Integrated Development Environment (IDE)

Install the IDE of your choice:

• Eclipse: https://eclipse.org/
• STS: https://spring.io/tools/sts/all
• IntelliJ: https://www.jetbrains.com/idea/
• VSCode: https://code.visualstudio.com/

Dev Container Environment

The project can be reopened inside a dev container in VSCode. This environment should have all the necessary dependencies to debug the ODE and its submodules. When attempting to run scripts in this environment, it may be necessary to make them executable with "chmod +x" first.

Checkstyle configuration

This project uses Checkstyle with a modified version of Google's Java Style guide to weakly enforce style standards. To configure Checkstyle with your chosen IDE follow one of the following guides. This repo's checkstyle configuration file can be found here. For a quick guide to Checkstyle, check out this short article.

• Intellij
• VSCode
• Eclipse

Back to top

6. Release History

Release Notes

Back to top

7. Contact Information

Contact the developers of the ODE application by submitting a Github issue.

Contact the ODE management representative using the information in the Code.gov Registration Info section.

License information

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either expressed or implied. See the License for the specific language governing permissions and limitations under the License.

Back to top

8. Contributing

Please read our contributing guide to learn about our development process, how to propose pull requests and improvements, and how to build and test your changes to this project.

Source Repositories - GitHub

• Main repository on GitHub (public)

<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0
                      http://maven.apache.org/xsd/settings-1.0.0.xsd">
<activeProfiles>     <activeProfile>github</activeProfile>   </activeProfiles>
<profiles>     <profile>       <id>github</id>       <repositories>         <repository>           <id>central</id>           <url>https://repo1.maven.org/maven2</url>         </repository>         <repository>           <id>github</id>           <url>https://maven.pkg.github.com/usdot-jpo-ode/jpo-ode</url>         </repository>       </repositories>     </profile>   </profiles>
<servers>     <server>       <id>github</id>       <username>${env.PACKAGEREADUSERNAME}</username>       <password>${env.PACKAGEREADTOKEN}</password>     </server>   </servers>
</settings>

<dependencies>
  <dependency>
    <groupId>usdot.jpo.ode</groupId>
    <artifactId>jpo-ode-core</artifactId>
    <version>1.0.0</version>
  </dependency>
  <dependency>
    <groupId>usdot.jpo.ode</groupId>
    <artifactId>jpo-ode-plugins</artifactId>
    <version>1.0.0</version>
  </dependency>
  <dependency>
    <groupId>usdot.jpo.ode</groupId>
    <artifactId>jpo-ode-common</artifactId>
    <version>1.0.0</version>
  </dependency>
  <dependency>
    <groupId>usdot.jpo.ode</groupId>
    <artifactId>jpo-ode-svcs</artifactId>
    <version>1.0.0</version>
  </dependency>
</dependencies>