The Experts below are selected from a list of 372 Experts worldwide ranked by ideXlab platform

Joe Celko - One of the best experts on this subject based on the ideXlab platform.

  • Chapter 11 – Thinking in SQL
    Joe Celko's Thinking in Sets, 2008
    Co-Authors: Joe Celko
    Abstract:

    Publisher Summary This chapter appraises the nature of programming in structured query language (SQL). Certain guidelines are offered to help programming SQL, namely—procedural code is replaced with declarative code, proprietary code is replaced with Standard SQL. DDL and DML are used together for a solution, and the solution shows a pattern that can be useful for similar problems. But perhaps the hardest thing to learn is thinking in sets. Sets can be defined two ways. Listing the elements; in math this is done with a pair of curvy brackets and a Comma-Separated List. This method is fine for small sets, and technically that is what a table is.The other method is to give a characteristic function that takes a value and returns a 1 or TRUE if the value is in the set and a zero or FALSE if it is not an element. That is what constraints do in SQL. Both these methods are used to define a table properly. A properly defined table is made up of one and only one kind of entity. The big leap in SQL programming is thinking in sets and not in process steps that handle one unit of data at a time. Phrases like “for the next x do…” poison the mental model of the problem. Programmers coming from procedural languages think in terms of actions. They add numbers, while declarative programmers look at a total.

  • Thinking in SQL
    Joe Celko's SQL Programming Style, 2005
    Co-Authors: Joe Celko
    Abstract:

    This chapter appraises the nature of programming in structured query language (SQL). Certain guidelines are offered to help programming SQL, namely—procedural code is replaced with declarative code, proprietary code is replaced with Standard SQL. DDL and DML are used together for a solution, and the solution shows a pattern that can be useful for similar problems. But perhaps the hardest thing to learn is thinking in sets. Sets can be defined two ways. Listing the elements; in math this is done with a pair of curvy brackets and a Comma-Separated List. This method is fine for small sets, and technically that is what a table is.The other method is to give a characteristic function that takes a value and returns a 1 or TRUE if the value is in the set and a zero or FALSE if it is not an element. That is what constraints do in SQL. Both these methods are used to define a table properly. A properly defined table is made up of one and only one kind of entity. The big leap in SQL programming is thinking in sets and not in process steps that handle one unit of data at a time. Phrases like “for the next x do…” poison the mental model of the problem. Programmers coming from procedural languages think in terms of actions. They add numbers, while declarative programmers look at a total.

Derek Weitzel - One of the best experts on this subject based on the ideXlab platform.

  • Scitokens/Xrootd-Scitokens: Flexible Authorization Handling
    2018
    Co-Authors: Brian Bockelman, Derek Weitzel
    Abstract:

    In this release, we significantly improve the authorization handling with the following three features: base_path can now take a Comma-Separated List of paths, allowing a single issuer to cover multiple parts of the filesystem namespace. restricted_path was introduced. This option restricts the paths the issuer is allowed to issue authorizations for within its base area(s). It is intended to ease the migrations to a SciTokens-based setup for existing storages where multiple groups share a same base area. default_user was introduced. This provides the ability to set the username in the credential for requests that pass the scitokens authorization. It allows the sysadmin to map the filesystem access of an issuer to a specific Unix username.

Clare Macrae - One of the best experts on this subject based on the ideXlab platform.

  • jgm/pandoc: pandoc 2.0
    2017
    Co-Authors: John Macfarlane, Jesse Rosenthal, Albert Krewinkel, Matthew Pickering, Nikolay Yakimov, Andrew Dunning, Mauro Bieg, Clare Macrae
    Abstract:

    New features New output format ms (groff ms). Complete support, including tables, math, syntax highlighting, and PDF bookmarks. The writer uses texmath's new eqn writer to convert math to eqn format, so a ms file produced with this writer should be processed with groff -ms -e if it contains math. New output format jats (Journal Article Tag Suite). This is an XML format used in archiving and publishing articles. Note that a URI-encoded CSL stylesheet (data/jats.csl) is added automatically unless a stylesheet is specified using --css. New output format gfm (GitHub-flavored CommonMark) (#3841). This uses bindings to GitHub's fork of cmark, so it should parse gfm exactly as GitHub does (excepting certain postprocessing steps, involving notifications, emojis, etc.). markdown_github has been deprecated in favor of gfm. New output format muse (Emacs Muse) (Alexander Krotov, #3489). New input format gfm (GitHub-flavored CommonMark) (#3841). This uses bindings to GitHub's fork of cmark. markdown_github has been deprecated in favor of gfm. New input format muse (Emacs Muse) reader (Alexander Krotov, #3620). New input format tikiwiki (TikiWiki markup) (rlpowell, #3800). New input format vimwiki (Vimwiki markup) (Yuchen Pei, #3705). Note that there is a new data file, data/vimwiki.css, which can be used to display the HTML produced by this reader and pandoc's HTML writer in the style of vimwiki's own HTML export. New input format creole (Creole 1.0) (#3994, Sascha Wilde). New syntax for Divs, with fenced_divs extension enabled by default (#168). This gives an attractive, plain-text way to create containers for block-level content. Added new syntax for including raw content in any output format, enabled by the raw_attribute extension (which is on by default for markdown and multimarkdown). The syntax is the same as for fenced code blocks or code inlines, only with {=FORMAT} for attributes, where FORMAT is the name of the output format (e.g., ms, html). Implement multicolumn support for slide formats (#1710). The structure expected is: contents... contents... Support has been added for beamer and all HTML slide formats. Allows line comments in templates, beginning with $-- (#3806). (Requires doctemplates 0.2.1.) Add --eol=crlf|lf|native flag and writer option to control line endings (Stefan Dresselhaus, #3663, #2097). Add --log option to save log messages in JSON format to a file (#3392). Add --request-header option, to set request headers when pandoc makes HTTP requests to fetch external resources. For example: --request-header User-Agent:blah. Added lua filters (Albert Krewinkel, #3514). The new --lua-filter option works like --filter but takes pathnames of special lua filters and uses the lua interpreter baked into pandoc, so that no external interpreter is needed. Note that lua filters are all applied after regular filters, regardless of their position on the Command line. For documentation of lua filters, see doc/lua-filters.md. Set PANDOC_READER_OPTIONS in environment where filters are run. This contains a JSON representation of ReaderOptions, so filters can access it. Support creation of pdf via groff ms and pdfroff. pandoc -t ms -o output.pdf input.txt. Support for PDF generation via HTML and weasyprint or prince (Mauro Bieg, #3909). pandoc -t html5 -o output.pdf --pdf-engine=prince. Added --epub-subdirectory option (#3720). This specifies the subdirectory in the OCF container that holds the EPUB specific content. We now put all EPUB related content in an EPUB/ subdirectory by default (later this will be configurable). mimetype META-INF/ com.apple.ibooks.display-options.xml container.xml EPUB/ fonts/ font.otf media/ cover.jpg fig1.jpg styles/ stylesheet.css content.opf toc.ncx text/ ch001.xhtml Added --resource-path=SEARCHPATH Command line option (#852). SEARCHPATH is Separated by the usual character, depending on OS (: on unix, ; on windows). Default resource path is just working directory. However, the working directory must be explicitly specified if the --resource-path option is used. Added –abbreviations=FILE option for custom abbreviations file (#256). Dfault abbreviations file (data/abbreviations) contains a List of strings that will be recognized by pandoc's Markdown parser as abbreviations. (A nonbreaking space will be inserted after the period, preventing a sentence space in formats like LaTeX.) Users can override the default by putting a file abbreviations in their user data directory (~/.pandoc on *nix). Allow a theme file as argument to --highlight-style. Also include a sample, default.theme, in data/. Allow --syntax-definition option for dynamic loading of syntax highlighting definitions (#3334). Lists in markdown by default now use the CommonMark variable nesting rules (#3511). The indentation required for a block-level item to be included in a List item is no longer fixed, but is determined by the first line of the List item. To be included in the List item, a block must be indented to the level of the first non-space content after the List marker. Exception: if are 5 or more spaces after the List marker, then the content is interpreted as an indented code block, and continuation paragraphs must be indented two spaces beyond the end of the List marker. See the CommonMark spec for more details and examples. Documents that adhere to the four-space rule should, in most cases, be parsed the same way by the new rules. Here are some examples of texts that will be parsed differently: - a - b will be parsed as a List item with a subList; under the four-space rule, it would be a List with two items. - a code Here we have an indented code block under the List item, even though it is only indented six spaces from the margin, because it is four spaces past the point where a continuation paragraph could begin. With the four-space rule, this would be a regular paragraph rather than a code block. - a code Here the code block will start with two spaces, whereas under the four-space rule, it would start with code. With the four-space rule, indented code under a List item always must be indented eight spaces from the margin, while the new rules require only that it be indented four spaces from the beginning of the first non-space text after the List marker (here, a). This change was motivated by a slew of bug reports from people who expected Lists to work differently (#3125, #2367, #2575, #2210, #1990, #1137, #744, #172, #137, #128) and by the growing prevalance of CommonMark (now used by GitHub, for example). Those who prefer the old behavior can use -f markdown+four_space_rule. Added four_space_rule extension. This triggers the old pandoc parsing rule for content nested under List items (the "four space rule"). Added spaced_reference_links extension (#2602). It allows whitespace between the two parts of a reference link: e.g. [a] [b] [b]: url This was previously enabled by default; now it is now forbidden by default. Add space_in_atx_header extension (#3512). This is enabled by default in pandoc and GitHub markdown but not the other flavors. This requirse a space between the opening #'s and the header text in ATX headers (as CommonMark does but many other implementations do not). This is desirable to avoid falsely capturing things ilke #hashtag or #5 Add sourcefile and outputfile template variables (Roland Hieber, #3431). Allow ibooks-specific metadata in epubs (#2693). You can now have the following fields in your YAML metadata, and it will be treated appropriately in the generated EPUB: ibooks: version: 1.3.4 specified-fonts: false ipad-orientation-lock: portrait-only iphone-orientation-lock: landscape-only binding: true scroll-axis: vertical Behavior changes Reader functions no longer presuppose that CRs have been stripped from the input. (They strip CRs themselves, before parsing, to simplify the parsers.) Added support for translations (localization) (#3559). Currently this only affects the LaTeX reader, for things like \figurename. Translation data files for 46 languages can be found in data/translations. Make --ascii work with DocBook output too. Rename --latex-engine to --pdf-engine, and --latex-engine-opt to --pdf-engine-opt. Removed --parse-raw and readerParseRaw. These were confusing. Now we rely on the +raw_tex or +raw_html extension with latex or html input. Thus, instead of --parse-raw -f latex we use -f latex+raw_tex, and instead of --parse-raw -f html we use -f html+raw_html. With --filter R filters are now recognized, even if they are not executable (#3940, #3941, Andrie de Vries). Support SVG in PDF output, converting with rsvg2pdf (#1793). Make epub an alias for epub3, not epub2. Removed --epub-stylesheet; use --css instead (#3472, #847). Multiple stylesheets may be used. Stylesheets will be taken both from --css and from the stylesheet metadata field (which can contain either a file path or a List of them). --mathml and MathML in HTMLMathMethod no longer take an argument. The argument was for a bridge JavaScript that used to be necessary in 2004. We have removed the script already. --katex improvements. The latest version is used, and the autoload script is loaded by default. Change MathJax CDN default since old one is shutting down (#3544). Note: The new URL requires a version number, which we'll have to update manually in subsequent pandoc releases in order to take advantage of mathjax improvements. --self-contained: don't incorporate elements with data-external="1" (#2656). You can leave an external link as it is by adding the attribute data-external="1" to the element. Pandoc will then not try to incorporate its content when --self-contained is used. This is similar to a feature already supported by the EPUB writer. Allow --extract-media to work with non-binary input formats (#1583, #2289). If --extract-media is supplied with a non-binary input format, pandoc will attempt to extract the contents of all linked images, whether in local files, data: uris, or external uris. They will be named based on the sha1 hash of the contents. Make papersize: a4 work regardless of the case of a4. It is converted to a4 in LaTeX and A4 in ConTeXt. Make east_asian_line_breaks affect all readers/writers (#3703). Underlined elements are now treated consistently by readers (#2270, hftf); they are always put in a Span with class underline. This allows the user to treat them differently from other emphasis, using a filter. Docx, Org, Textile, Txt2Tags, and HTML readers have been changed. Improved behavior of auto_identifiers when there are explicit ids (#1745). Previously only autogenerated ids were added to the List of header identifiers in state, so explicit ids weren't taken into account when generating unique identifiers. Duplicated identifiers could result. This simple fix ensures that explicitly given identifiers are also taken into account. Use table-of-contents for contents of toc, make toc a boolean (#2872). Changed markdown, rtf, and HTML-based templates accordingly. This allows you to set toc: true in the metadata; this previously produced strange results in some output formats. For backwards compatibility, toc is still set to the toc contents. But it is recommended that you update templates to use table-of-contents for the toc contents and toc for a boolean flag. Change behavior with binary format output to stdout. Previously, for binary formats, output to stdout was disabled unless we could detect that the output was being piped (and not sent to the terminal). Unfortunately, such detection is not possible on Windows, leaving windows users no way to pipe binary output. So we have changed the behavior in the following way: Output to stdout is allowed when it can be determined that the output is being piped (on non-Windows platforms). If the -o option is not used, binary output is never sent to stdout by default; instead, an error is raised. If -o - is used, binary output is sent to stdout, regardless of whether it is being piped. This works on Windows too. Better error behavior: uses of error have been replaced by raising of PandocError, which can be trapped and handled by the calling program. Removed hard_line_breaks extension from markdown_github (#3594). GitHub has two Markdown modes, one for long-form documents like READMEs and one for short things like issue coments. In issue comments, a line break is treated as a hard line break. In README, wikis, etc., it is treated as a space as in regular Markdown. Since pandoc is more likely to be used to convert long-form documents from GitHub Markdown, -hard_line_breaks is a better default. Include backtick_code_blocks extension in mardkown_mmd (#3637). Escape MetaString values (as added with -M/--metadata flag) (#3792). Previously they would be transmitted to the template without any escaping. Note that --M title='*foo*' yields a different result from --- title: *foo* --- In the latter case, we have emphasis; in the former case, just a string with literal asterisks (which will be escaped in formats, like Markdown, that require it). Allow em, cm, in for image height/width in HTML, LaTeX (#3450). HTML writer: Insert data- in front of unsupported attributes. Thus, a span with attribute foo gets written to HTML5 with data-foo, so it is valid HTML5. HTML4 is not affected. This will allow us to use custom attributes in pandoc without producing invalid HTML. (With help from Wandmalfarbe, #3817.) Plain writer: improved super/subscript rendering. We now handle more non-digit characters for which there are sub/superscripted unicode characters. When unicode sub/superscripted characters are not available, we use _(..) or ^(..) (#3518). Docbook, JATS, TEI writers: print INFO message when omitting interior header (#3750). This only applies to section headers inside List items, e.g., which were otherwise silently omitted. Change to --reference-links in Markdown writer (#3701). With --reference-location of section or block, pandoc will now repeat references that have been used in earlier sections. The Markdown reader has also been modified, so that exactly repeated references do not generate a warning, only references with the same label but different targets. The idea is that, with references after every block, one might want to repeat references sometimes. ODT/OpenDocument writer: Support lang attribute (#1667). Added support for --toc (#2836). Thanks to @anayrat. Docx writer: lang meta, see #1667 (Mauro Bieg, #3515). Change FigureWithCaption to CaptionedFigure (iandol, #3658). Use Table rather than Table Normal for table style (#3275). Table Normal is the default table style and can't be modified. Pass through comments (#2994). We assume that comments are defined as parsed by the docx reader: I want I left a comment.some text to have a comment on it. We assume also that the id attributes are unique and properly matched between comment-start and comment-end. Bookmark improvements. Bookmark start/end now surrounds content rather than preceding it. Bookmarks generated for Div with id (jgm/pandoc-citeproc#205). Add keywords metadata to docx document properties (Ian). RST writer: support unknown interpreted text roles by parsing them as Span with role attributes (#3407). This way they can be manipulated in the AST. HTML writer: Line block: Use class instead of style attribute (#1623). We now issue and include a default definition for line-block in the default templates, instead of hard-coding a style on the div. Add class footnoteBack to footnote back references (Timm Albers). This allows for easier CSS styling. Render SmallCaps as span with smallcaps class (#1592), rather than using a style attribute directly. This gives the user more flexibility in styling small caps in CSS. With reveal.js we use data-src instead of src for images for lazy loading. Special-case .stretch class for images in reveal.js (#1291). Now in reveal.js, an image with class stretch in a paragraph by itself will stretch to fill the whole screen, with no caption or figure environment. Added warnings for non-rendered blocks to writers. Writers now raise an error on template failure. When creating a PDF via LaTeX, warn if the font is missing some characters (#3742). Remove initial check for PDF-creating program (#3819). Instead, just try running it and raise the exception if it isn't found at that point. This improves things for users of Cygwin on Windows, where the executable won't be found by findExecutable unless .exe is added. The same exception is raised as before, but at a later point. Readers issue warning for duplicate header identifiers (#1745). Autogenerated header identifiers are given suffixes so as not to clash with previously used header identifiers. But they may still coincide with an explicit identifier that is given for a header later in the document, or with an identifier on a div, span, link, or image. We now issue a warning in this case, so users can supply an explicit identifier. CommonMark reader now supports emoji, hard_line_breaks, smart, and raw_html extensions. Markdown reader: Don't allow backslash + newline to affect block structure (#3730). Note that as a result of this change, the following, which formerly produced a header with two lines Separated by a line break, will now produce a header followed by a paragraph: # Hi there This may affect some existing documents that relied on this undocumented and unintended behavior. This change makes pandoc more consistent with other Markdown implementations, and with itself (since the two-space version of a line break doesn't work inside ATX headers, and neither version works inside Setext headers). Org reader (Albert Krewinkel, unless noted): Support table.el tables (#3314). Support macros (#3401). Support the #+INCLUDE: file inclusion mechanism (#3510). Recognized include types are example, export, src, and normal org file inclusion. Advanced features like line numbers and level selection are not implemented yet. Interpret more meta value as inlines. The values of the following meta variables are now interpreted using org-markup instead of treating them as pure strings: keywords (Comma-Separated List of inlines), subtitle (inline values), nocite (inline values, can be repeated). Support \n export option (#3940). This turns all newlines in the text into hard linebreaks. RST reader: Improved admonition support (#223). We no longer add an admonition class, we just use the class for the type of admonition, note for example. We put the word corresponding to the label in a paragraph inside a Div at the beginning of the admonition with class admonition-title. This is about as close as we can get to RST's own output. Initial support of .. table directive. This allows adding captions to tables. Support .. line-block directive. This is deprecated but may still be in older documents. Support scale and align attributes of images (#2662). Implemented implicit internal header links (#3475). Support RST-style citations (#853). The citations appear at the end of the document as a definition List in a special div with id citations. Citations link to the definitions. Recurse into bodies of unknown directives (#3432). In most cases it's better to preserve the content than to emit it. This isn't guaranteed to have good results; it will fail spectacularly for unknown raw or verbatim directives. Handle chained link definitions (#262). For example, .. _hello: .. _goodbye: example.com Here both hello and goodbye should link to example.com. Support anchors (#262). E.g. `hello` .. _hello: paragraph This is supported by putting "paragraph" in a Div with id hello. Support :widths: attribute for table directive. Implement csv-table directive (#3533). Most attributes are supported, including :file: and :url:. Support unknown interpreted text roles by parsing them as Span with "role" attributes (#3407). This way they can be manipulated in the AST. HTML reader: parse a span with class smallcaps as SmallCaps. LaTeX reader: Implemented \graphicspath (#736). Properly handle column prefixes/suffixes. For example, in \begin{tabular}{>{$}l{$}l{$}l

Vetterli Adrien - One of the best experts on this subject based on the ideXlab platform.

  • LiDAR-derived forest structure data and predictions of the locations of old-growth forests for Central Finland.
    2016
    Co-Authors: Happonen Konsta, Maanavilja Liisa, Vetterli Adrien
    Abstract:

    INTRO This archive contains data and analysis code for the Biodiversity Map -project conducted by Open Knowledge Finland (http://fi.okfn.org/projects/biodiversity-map/) LICENCE The files Listed below are all released to the public domain under a CC0 public domain dedication (https://creativecommons.org/publicdomain/zero/1.0/) FILE DESCRIPTIONS FILE 1: background.zip Inside the archive is a Comma-Separated file "background.csv" containing LiDAR-derived forest structure variables for 2/3 of Central Finland. These were derived from 3 raster data sets describing forest canopy maximum height (mh), forest canopy cover (cc) and lidar return intensity (in). The rasters had resolutions of 6 metres, 6 metres and 2 metres, respectfully. An 18 m resolution grid was then used to aggregate the rasters into average, minimum and maximum values + standard deviations of the original variables. The original LiDAR data was made available by the National Land Survey of Finland. FILE 2: conservation.lambdas This file contains fitted parameters for the maxent model. For more information, check maxent documentation at https://www.cs.princeton.edu/~schapire/maxent/ FILE 3: conserved_swd.csv Forest structure variables at 18 meter resolution for old-growth conservation areas in Central Finland. A subset of background.csv. This file still has a header, the variables are the same as in background.csv FILE 4: grass_create_forest_rasters_from_las.sh A shell script used to convert LiDAR files to raster maps of forest structure with GRASS 7. FILE 5: lidar_coverage.png A map showing the extent of LiDAR data available for Central Finland when we did the analyses. FILE 6: maxent_model_run_product.sh A shell script used to fit the maximum entropy model to predict the locations of conservation-area-like forests in Central Finland. FILE 7: projection_product.csv The results of the maxent model in a Comma Separated file. The first row has the variable names: x,y,product_fit. x and y are coordinates in the CRS ETRS-TM35FIN (EPSG:3067). product_fit is "the probablility that this 18*18 meter grid cell is old-growth conservation area". FILE 8: README A file with a description of the dataset in human-readable form. VALIDATION FILES The data in these files was collected to validate the results of the aforementioned maxent model. The data were collected in a hierarchical sampling scheme: six randomly determinded unintersecting 9 km * 9 km landscape windows were chosen for sampling. From each window, three samples were taken. One sample from conservation areas, one sample from the "best" 10 % of forests as determined by the maxent model excluding conservation areas and one random sample. Not all windows contained conservation areas, and not all areas were accessible (islands, for example). In addition a few areas were skipped due to time constraints. The sampled points are identified by their lanscape window (suuralue), their sample (otos) and their sample number (mittauspiste). FILE 9: validation_felled.csv A Comma Separated List of those points that were not measured because they were felled. FILE 10: validation_gps_results_2016-09-07.csv A List of gps coordinates for all the sample points. product_fit is the value of the geographically closest prediction from the maxent model described above. FILE 11: validation_lying_deadwood_transects_2016-08-30.csv A Comma Separated file with data from deadwood transects. From each validation point, three 30 m long transects were made with 120 degree angles between them, and all lying deadwood more than 2 cm in diameter were measured. For some validation points, there were geographical obstructions which prevented the full 90 m of transect being surveyed, this is also recorded in the data. Each row holds measurements from one lying trunk. FILE 12: validation_relascope_2016-08-30.csv Relascope measurements from the validation points. Each row is measurements for one species from one validation point. Dead and alive trees are counted separately. MORE INFORMATION For more in-depth descritions of the files, read the file named README. For some auxilliary files and information, check our old hackathon repository on github: https://github.com/Koalha/bdm_hackatho

Brian Bockelman - One of the best experts on this subject based on the ideXlab platform.

  • Scitokens/Xrootd-Scitokens: Flexible Authorization Handling
    2018
    Co-Authors: Brian Bockelman, Derek Weitzel
    Abstract:

    In this release, we significantly improve the authorization handling with the following three features: base_path can now take a Comma-Separated List of paths, allowing a single issuer to cover multiple parts of the filesystem namespace. restricted_path was introduced. This option restricts the paths the issuer is allowed to issue authorizations for within its base area(s). It is intended to ease the migrations to a SciTokens-based setup for existing storages where multiple groups share a same base area. default_user was introduced. This provides the ability to set the username in the credential for requests that pass the scitokens authorization. It allows the sysadmin to map the filesystem access of an issuer to a specific Unix username.