The simplest authenticated pubsub system. Send messages through user-defined topics (aka channels). By default, topics are private to the authenticated ssh user. The default pubsub model is multicast with bidirectional blocking, meaning a publisher (pub) will send its message to all subscribers (sub) for a topic. There can be many publishers and many subscribers on a topic. Further, both pub and sub will wait for at least one event to be sent or received on the topic.

pipe is a simple and secure way of putting together composable directional streams of data, just like a *nix | operator!

Features #

• Zero-install
• Familiar *nix pipes API
• Authenticated pubsub using ssh
• Private pubsub by default
• Public pubsub by topic (opt-in)
• Multicast (many pubs to many subs)
• Bidirectional (e.g. chat)
• Paradigms for connecting to a topic:
  • Read (sub)
  • Write (pub)
  • Read & Write (pipe)

Use cases #

• Send desktop notifications
• File sharing
• Pipe command output
• Chat
• Reverse shell
• CI/CD

Examples #

For example, maybe you have a *nix pipe you're using on the command line like so:

1tail -f -n 0 /tmp/foo.log | grep --line-buffered "ERROR" | xargs -I{} -L1 osascript -e 'display notification "{}" with title "Pipe Notification"'

This simple one liner will grab new messages from a log file, check if it contains ERROR and then send a notification for macOS using AppleScript. This works if you run your app locally, but what if you run the app on a remote server? You can use pipe to connect remote and local without ever leaving the command line!

On your remote side, you would start a tail and pub it to a topic:

1tail -f -n 0 /tmp/foo.log | ssh pipe.pico.sh pub foo.log

On your local side, you would sub it to the notify command:

1ssh pipe.pico.sh sub foo.log | grep --line-buffered "ERROR" | xargs -I{} -L1 osascript -e 'display notification "{}" with title "Pipe Notification"'

The beauty of this method is that the command will wait until the sub is started before any data is consumed, ensuring you never miss a log line. If you didn't want it to wait for a sub, you can add -b=false (b for blocking) to the pub to prevent it from blocking.

Once you stop the pub command, the sub will also exit. You can also prevent this by adding -k (k for keepalive) to the sub command. With both blocking disabled and keepalive enabled, the commands would look like so:

Remote:

1tail -f -n 0 /tmp/foo.log | ssh pipe.pico.sh pub -b=false foo.log

Local:

1ssh pipe.pico.sh sub -k foo.log | grep --line-buffered "ERROR" | xargs -I{} -L1 osascript -e 'display notification "{}" with title "Pipe Notification"'

We can combine this with any commands we want and create a pretty robust pub/sub system. We can even send full command output through a pipe:

1ssh pipe.pico.sh sub htop

1htop | ssh pipe.pico.sh pub htop

Now what if you wanted to have bi-directional IO? That's where our last command of Pipe comes in, pipe! Pipe allow you to open a bi-directional client on any topic. It's fully non-blocking and can allow you to do things like have fully interactive chat over pipe. For example, running the following in two terminals:

1ssh pipe.pico.sh pipe chat

Will allow you to type and read messages as if you were sitting at the same terminal!

CLI #

We have a bunch of demo examples on the main pipe website so be sure to check those out.

 1~$ ssh pipe.pico.sh help
 2Command: ssh pipe.pico.sh <help | ls | pub | sub | pipe> <topic> [-h | args...]
 3
 4The simplest authenticated pubsub system.  Send messages through
 5user-defined topics.  Topics are private to the authenticated
 6ssh user.  The default pubsub model is multicast with bidirectional
 7blocking, meaning a publisher ("pub") will send its message to all
 8subscribers ("sub").  Further, both "pub" and "sub" will wait for
 9at least one event to be sent or received. Pipe ("pipe") allows
10for bidirectional messages to be sent between any clients connected
11to a pipe.
12
13Think of these different commands in terms of the direction the
14data is being sent:
15
16- pub => writes to client
17- sub => reads from client
18- pipe => read and write between clients

Subs #

1~$ ssh pipe.pico.sh sub -h
2Usage: sub <topic> [args...]
3Args:
4  -a string
5        Comma separated list of pico usernames or ssh-key fingerprints to allow access to a topic
6  -c    Don't send status messages
7  -k    Keep the subscription alive even after the publisher has died
8  -p    Subscribe to a public topic

Pubs #

 1~$ ssh pipe.pico.sh pub -h
 2Usage: pub <topic> [args...]
 3Args:
 4  -a string
 5        Comma separated list of pico usernames or ssh-key fingerprints to allow access to a topic
 6  -b    Block writes until a subscriber is available (default true)
 7  -c    Don't send status messages
 8  -e    Send an empty message to subs
 9  -p    Publish message to public topic
10  -t duration
11        Timeout as a Go duration to block for a subscriber to be available. Valid time units are 'ns', 'us' (or 'µs'), 'ms', 's', 'm', 'h'. Default is 30 days. (default 720h0m0s)

Pipes #

1~$ ssh pipe.pico.sh pipe -h
2Usage: pipe <topic> [args...]
3Args:
4  -a string
5        Comma separated list of pico usernames or ssh-key fingerprints to allow access to a topic
6  -c    Don't send status messages
7  -p    Pipe to a public topic
8  -r    Replay messages to the client that sent it

Web Interface #

Now what if you don't have a terminal available? Not a problem! Pipe has a web component that works side by side, pipe-web. For example, let's start a notification sub through the terminal like so (not p for public, so anyone can send us a notification):

1ssh pipe.pico.sh sub -p -k notifications | xargs -I{} -L1 osascript -e 'display notification "{}" with title "Pipe Notification"'

We can send a POST to the pipe-web service to send a message onto that topic like so:

1echo "Hello world" | curl -X POST https://pipe.pico.sh/topic/notifications --data-binary @-

And a notification will pop up! Now it's important to note that this is risky, anyone can use a "public" topic (on both the terminal or pipe-web). We can make this less risky by starting the notifications topic with an access list set. And if we want pipe-web to access it, we need to provide "pico" to the access list setting:

1ssh pipe.pico.sh sub -a pico -p -k notifications | xargs -I{} -L1 osascript -e 'display notification "{}" with title "Pipe Notification"'

Now, only pipe-web and yourself are able to access this public topic.

pipe-web comes with a few caveats, namely all topics need to be public for it to work. You can set an access list on the topic, but pipe-web is an unauthenticated service. Therefore, anyone can send a post request to the process.

WebSockets #

Just like you can use pipe-web for GET and POST requests, you can also use WebSockets to get a bi-directional stream to a topic. This functionality is most closely related to the pipe CLI option. You can try this functionality here. The API is simple and documented below. This example terminal also accepts query params like the following:

Open a pipe #

Subscribe to a topic #

Publish to a topic #

Subscribe to a topic #

Publish to a topic #

pipemgr #

pipemgr is a docker image that will listen for logs from other running containers and pipe their logs through a topic called container-drain.

 1services:
 2  pipemgr:
 3    build:
 4      context: .
 5    image: ghcr.io/picosh/pipemgr:latest
 6    restart: always
 7    volumes:
 8      - /var/run/docker.sock:/var/run/docker.sock:ro
 9      - $HOME/.ssh/id_ed25519:/key:ro
10    healthcheck:
11      test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
12      interval: 2s
13      timeout: 5s
14      retries: 5
15      start_period: 1s
16  httpbin:
17    image: kennethreitz/httpbin
18    command: gunicorn -b 0.0.0.0:3000 httpbin:app -k gevent --access-logfile -
19    ports:
20      - 3000:3000
21    labels:
22      pipemgr.enable: true
23      # filter log lines with:
24      # pipemgr.filter: "GET.+(404)"
25    depends_on:
26      pipemgr:
27        condition: service_healthy

Caveats #

You must always pipe something into pub or else it will block indefinitely until the process is killed. However, you can provide a flag to send an empty message: pub topic -e.

Inspiration #

A special thanks to patchbay.pub for our inspiration.

Latest posts #

2024-10-06 pipe: our pubsub ssh service