Building packages for 3rd party code with CI/CD in GitHub or GitLab.

Besides building application packages, you might want to create your own (custom) OS packages – for example in rpm or deb format. Which can then be tested and deployed. Using GitHub Actions or GitLab’s CI & API, creating packages can easily be automated. And that’s what this post is about.

First make sure we get source code. GitLab supports Repository mirroring and GitHub has a checkout action and several actions available in Marketplace that fully mirror repos. Also if there is no method yet to build the code and create a package, we need to create our own build script (in a new branch). Now a build job can run, either on a runner or using build image and container.

GitHub

Add job to workflow to get build artifacts and create release using ‘gh’ cli tool

Example:

jobs:
# ...
   create-release:
     name: Get artifacts and create release
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v4
       - name: Download artifacts
         uses: actions/download-artifact@v4
         with:
           path: build-artifacts
       - name: Create release using 'gh'
          run: |
            gh release create mypkg-v$(date +%Y%m%d) $(find -type f -printf '%p ') --notes "my release  notes"
         working-directory: ./build-artifacts
         env:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

GitLab

Ensure build artifacts are stored earlier in your pipeline and add a job that uploads and publish to package registry

Example:

# ...
build_job:
  stage: build
  #...
  artifacts:
      name: my-build-artifacts
      paths:
        - "build/example*"

publish_job:
  stage: publish
  dependencies: build_job
  script:
    - '/usr/bin/curl --silent
      --header "JOB-TOKEN: $CI_JOB_TOKEN"
      --upload-file "$FILE_NAME"
      "https://instance-or-gitlab.com/projects/${CI_PROJECT_ID}/packages/generic/my_pkg/$(date +%Y%m%d)/${FILE_NAME}?status=default" 
      grep -q "201 Created" || exit 1'

For “versioning” we’ve simply used the current date, this probably should be a git tag or extracted from source. The next steps are adding a job to test the package and to deploy it (push). To be able to pull instead, you could set up your own repository. You could also consider using SaaS like packagecloud.io, artifactory or Ubuntu’s PPA to distribute the package.

IDE/Editor

• Eclipse (CDT)

• VS Code (MS C extensions)

• Vim

Formatting:

1TBS of course ;)

Setup vi manually:

set noai tabstop=8 shiftwidth=8 softtabstop=8 noexpandtab

or use modeline in file:

/* vim: set noai tabstop=8 shiftwidth=8 softtabstop=8 noexpandtab: */

Strings

char greeting[6] = {'H', 'e', 'l', 'l', 'o', '\0'};
char str[12] = "Hello World";
char str[0] = '\0';
char buf[];

copy: strcpy(str2, str1)
concat: strcat(str1, str2)
length: strlen(str)
duplicate: ptr = strdup(str)
substring: strstr(str1, str2)
extract: strsep(*strptr, delimstr)

strstr():

locate subscring

or: “check if ‘sent’ contains ‘word'”

if(strstr(sent, word) != NULL) //..or skip NULL check

strsep():

• extract token
• or: “split substring using delim”
• like deprecated strtok()

while((found = strsep(&string,",")) != NULL)

Things to remember

• set it before using it (initialize)
• strings are char arrays terminated by \0
• strings in double quotes are auto terminated by \0
• strcpy : watch out for buffer overruns with long strings
• use : checks, char buf[1024], \0 term, malloc(strlen(str) + xx)
• buf[] has no size and no space allocated
• convert int to string: itoa, sprintf, snprintf(buf, 10, "%d", num);

tutorialspoint.com/cprogramming/c_strings.htm

Print

printf outputs to stdout stream
fprintf goes to a file handle FILE*
sprintf goes to a buffer you allocated (char*)

printf("This is a str: %s\n", str);
fprintf(fh, "log: %s", str);
sprintf(buf, "%s %s", str1, str2);

int: %d
long int: %ld
long long int: %lld
unsigned long long int: %llu

“%s” format specifier for printf always expects a char* argument

Pointers

// ptr is going to store address of integer value
int i = 20;
int *ptr;

                Operator        Operator Name           Purpose
*ptr            *               Value at Operator       gives Value stored at particular address
type* ptr;                      Same                    Same
&ptr            &               Address Operator        gives Address of Variable
**ptr                           Double Pointer          declares a Pointer to a Pointer

• using *ptr is actually the first element (not its address)
• while int *var is the same as int* var, *var is better/more clear to use
• initialize (NULL or valid address)

Extern

• tell compiler variable is declared elsewhere
• use only in one place

example.h:

extern int global_variable;

example.c:

#include "example.h"
int global_variable = 1337;

example2.c:

#include "example.h"
void use_it(void)
{
    printf("Global variable: %d\n", global_variable++);
}

Compiler messages

warning: implicit declaration of function 'function' [-Wimplicit-function-declaration]

You are using a function for which the compiler has not seen a declaration (“prototype”) yet.

int main()
{
    fun(2, "21"); /* The compiler has not seen the declaration. */
    return 0;
}

int fun(int x, char *p)
{
    /* ... */
}

You need to declare your function before main, like this, either directly or in a header:

int fun(int x, char *p);

From stackoverflow.com/a/8440833

Macros

ifdef : “if defined” (quelle surprise;)

#ifndef : if NOT defined

example.c:

#ifdef DEBUG
 printf("DEBUG: %s\n", dbgstuff);
#endif

gcc -DDEBUG -c

gcc.gnu.org/onlinedocs/cpp/Ifdef.html

Include guard

#ifndef : checks whether the given token has been #defined earlier in the file or in an included file; if not, it includes the code between it and the closing #else or, if no #else is present, #endif statement.

#ifndef : is often used to make header files idempotent by defining a token once the file has been included and checking that the token was not set at the top of that file.

example.h:

#ifndef _INCL_GUARD
#define _INCL_GUARD
#endif

cprogramming.com/reference/preprocessor/ifndef.html

Malloc

char *buf = (char *) malloc(bufferSize);

where bufferSize is the runtime result of some computation

Examples:

char *string = (char*)malloc(n+1 * sizeof(char));
size_t length = strlen(str1) + strlen(str2) + 1;
char *concat = malloc(sizeof(char) * length);
char* buffer = (char*)malloc(256);
rhost = (char *)malloc(sizeof(char)*128);

• if allocatable buffer space is variable, use malloc instead of buf[1024]
• when done use free: free(buf);
• strdupe already does malloc, no malloc needed

Execute command

exec() & friends like execve

system("ls -la")

get output: popen()

FILE *fp = popen("ls -la", "r");
if (fp)
 fscanf(fp, "%100s", var);

-or-

 fgets(var, 100, fp);

Return

return 0 or 1:

int function {
 if (success) { return 0; }
 return 1;
}

return char* value:

char* function {
 char *buf;
 ...
 return buf;
}

(use malloc or buf[1024] and free)

Examples:

str functions (example):

printf ("\nDEBUG: strncmp %s %i\n", flist_getfilename(ftmp), strncmp(flist_getfilename(ftmp), ".mp3", 4));
printf ("\nDEBUG: strcmp %s %i\n", flist_getfilename(ftmp), strcmp(flist_getfilename(ftmp), ".mp3"));
printf ("\nDEBUG: strcasecmp %s %d\n", flist_getfilename(ftmp), strcasecmp(flist_getfilename(ftmp), ".mp3"));

more strcmp:

if (!strncmp(flist_getfilename(ftmp), ".mp3", 4))
if (strncmp(flist_getfilename(ftmp), ".mp3", 4) != 4 )

Print modules:

python3 -c "import sys;print (sys.path)"
python3 -c "import sys;print (sys.modules)"
python3 -c "help('modules')"
python3 -c "help('modules package')"
pydoc3 modules

Packages:

• Location in Debian: /usr/lib/python{3}/dist-packages/pkgname/{__init__.py}
• Other distro’s use “site-packages”

pip3 install pkgname
pip3 uninstall pkgname
pip3 freeze
python3 -m pip install pkgname

python3
import pip
sorted(["%s==%s" % (i.key, i.version) for i in pip.get_installed_distributions()])

Virtualenv:

$ pip3 install virtualenv
$ mkdir workdir && cd workdir
$ virtualenv project
$ source project/bin/activate
$ pip3 install requests

(and/or git clone + python3 setup.py install / etc)

$ deactivate
$ rm -rf project

Strings:

Besides %-format  and str.format() there’s now also f-Strings (since Python 3.6).

Example:

foo="value"

print(f'This string has a variable: {foo}.')

Notes:

• make sure your script.py does not have the same name as package
• print syntax changed from python2 to 3: print ("foo")

List installed modules:

Pick one :)

• perl -V
• perldoc <module name>
• perldoc perllocal
• instmodsh
• cpan -[l,a]

or:

perl -e 'use ExtUtils::Installed; foreach (ExtUtils::Installed->new()->modules()) {print $_ . "\n";}'

Install modules in homedir:

• install “local::lib” module
• print settings: perl -Mlocal::lib
• add to .bashrc: eval "$(perl -I$HOME/perl5/lib/perl5 -Mlocal::lib)"

Then there’s also “Perlbrew“.