Reasons for failures included an unhandled error code, erroneous sorting
assumptions, and a broken computation of the next insert ID in tests
Five failures remain.
This involved changes to the driver interface as well as the database
schemata. The most significantly altered queries were for article
selection and marking, which relied upon unusual features of SQLite.
Overall query efficiency should not be adversely affected (it may have
even imprved) in the common case, while very rare cases (not presently
triggered by any REST handlers) require more queries.
One notable benefit of these changes is that functions which query
articles can now have complete control over which columns are returned.
This has not, however, been implemented yet: symbolic column groups are
still used for now.
Note that PostgreSQL still fails many tests, but the test suite runs to
completion. Note also that one line of the Database class is not
covered; later changes will eventually make it easier to cover the line
in question.
PDO does not adequately inform PostgreSQL of a parameter's type, so type
casts are required. Rather than adding these to each query manually, the
queries are instead processed to add type hints automatically.
Unfortunately the queries are processed rather naively; question-mark
characters in string constants, identifiers, regex patterns, or geometry
operators will break things spectacularly.
While the test suite passes, this commit yields a broken server: replacing ad hoc request objectss with PSR-7 ones is still required, as is emission of PSR-7 responses. Both will come in subsequent commits, with tests
Diactoros was chosen specifically because it includes facilities for emitting responses, something which is awkward to test. The end of this refactoring should see both the Response and Request classes disappear, and the general REST class fully covered (as well as any speculative additions to AbstractHanlder).
- Changed 'transform' method to use ValueInfo throughout. This fixes a number of obscure bugs
- Changed the 'add' and 'sub' methods to default to "now" rather than null. This means null passes through rather than being interpreted as the current time, to be consistent with other date tools
- Also changed the 'add' and 'sub' methods so that they operate correctly with invalid date strings
- Added tests for the class; improves #66
- Modified TTRSS tests because the "iso8601" format string in ValueInfo is different from Date's older format
Also move date formats to the ValueInfo class
Standardizing on immutables avoids any possible ambiguity in the API of the resultant value, as well as any ambiguity as to whether a DateTime output instance is the same instance or a clone (they had been clones)
This sees the addition of a dateOutFormat parameter to ValueInfo::normalize(), as well as a general simplification of how parameter binding works.
Some value type-casting results are slightly different, but this simply makes SQL statement objects consistent with the rest of the system.
The changes in this commit should make it more practical to:
- Allow the driver to decide for itself whether to try creating a PDO object if its own requirements are not met
- Have any driver use a generic schema update procedure
- Use the same constructor for native and PDO SQLite
Code coverage information is now gathered via phpdbg (a separate executable) by if available rather than xdebug, as the latter is hard to turn on and off.
A "test:quick" task has also been added to Robo, which excludes 31 tests which together account for almost two thirds of the test run time. This should pave the way for testing to be added as a commit hook for Git.
Though TTRSS itself (usually) omits items that have a counter of zero, at least one client takes this to mean the last-seen counter is unchanged, rather than zero.
Real-world use suggests 5s is inadequate. Rather than bumping up small amounts as things break, it makes more sense to wait a very long time and investigate possible long-term solutions later, once logging is implemented.
Includes PHPDoc license tag in the file-level block with accompanying copyright notice.
Also added an AUTHORS file on the off chance of outside contributions
Includes PHPDoc license tag in the file-level block with accompanying copyright notice.
Also added an AUTHORS file on the off chance of outside contributions
This introduces a data model function of unusual privilege: it can retrieve favicon URLs for any subscription, regardless of user ID. This is a single-purpose hack and its use should be avoided if at all possible.
This required adding the "notes" column to the arsse_marks table and adding same as a target value in Database::articleMark()
The Context class was also adjusted to remove the possibility of false positives in some tests
Queries for multiple specific articles are limited in size because of limits on the number of bound query parameters.
Currently this limit is somewhat arbitrarily set at 50, but it may increase.
Historically controllers would be responsible for chunking input, but this will present problems when the expected output is a result set, and of course the maintenance burden increases as the number of controllers increases.
This commit transfers the burden to the data model, and consequently introduces a ResultAggregate class which collects chunked result sets (currently only for articleList).
In the course of making these changes the mock Result class was also largely rewritten, fixing many bugs with it.
This commit does not modify the controllers nor their tests; this will be done in a subsequent commit.
- Result sets are now single-use; this is required for PDO drivers (PDO result sets are not rewindable)
- Change savepoint exceptions to be simple database exceptions; codes remain the same
This has the side-effect of removing the ability to reset a feed's title by passing null explicitly. As a non-standard behaviour it was simpler to just remove it.
This method provides generalized, consistent type casting more versatile than PHP's basic type juggling while hiding the significant complexity in achieving this.
While this commit does not change any existing code to use the new method, the intent is for both API handlers and database drivers to use the same basic rules for type conversion while still allowing for differing failure modes.
Also consolidated article star counting into a generic articleCount function which accepts a context.
This may lead to slight efficiency losses in either listing or marking (and more significant ones in counting starred), but the advantages of centralized context handling are significant with the future addition of labels and the need to count articles under various future contexts in TTRSS.
- Backend functions for adding, listing, removing, and editing (renaming) labels currently implemented
- TTRSS functions for adding (fixes #96), removing (fixes #97), and renaming (fixes #98) labels currently implemented
- Database functions now accept any input, but throw typeViolation exceptions where appropriate instead of idMissing or subjectMissing
- Added unit tests for the new Misc\ValueInfo static class
- Added ValueInfo::id() method to centrally validate database IDs, and made use of it consistently
- Made use of PHP's filter_var() function where appropriate when validating or sanitizing input
- Made the NCN protocol handler reject most invalid IDs before handing off to method handlers
- Made NCN's feedUpdate and subscriptionMove methods return 422 on invalid input
- Adjusted several tests to handler type violations
- Specifying a non-integer parent no longer silently casts to 0 or 1
- Specifying a folder ID of 0 now always converts to null automatically
- Performing both a rename and move to root in the same operation no longer results in potential duplicates
- Calling folderSetProperties with an empty data array no peforms an update; it now returns false before the update call
- Modification timestamps are now actually updated when a folder is modified
- Constraint violation exceptions triggered by code (rather than the database) now print a message
- Renaming a folder or subscription to a non-string value (e.g. an array) throws an exception rather than silently casting
- Added tests to better cover all the above
- Centralized the normalization of integers and title strings into a new ValueInfo static class
TTRSS' error message for these cases (NOT_LOGGED_IN) is not especially helpful, but that's what it returns, so that's what we should return, albeit with correct HTTP status codes.
- Added a 'user add' function to the CLI
- Removed the last trace of paths from configuration; paths must default to null and Arsse must determine sensible defaults at runtime
- Don't load a config (and possibly create a database) in CLI if a configuration is not required
- Removed the 'dbSchemaBase' config option, which is really a testing hack
- Added sample Nginx configuration
- Fixed bug in REST handler
- Readme still needs work
Previously arsse_marks had a reference to arsse_users along with arsse_articles; the former has now changed to arsse_subscriptions.
Consequently deleting a subscription now deletes marks. Tests have been simplified as a consequence.
This change was understaken to simplify counting valid marks on articles for the purposes of article cleanup; now all marks are valid marks
- Basic update service handles only one feed at a time and possibly leaks memory
- Output for REST requests is still very basic
- No avatar support
- No reporting of whether cron works
- No cleanup before or after feed updates
- use DateFormatter throughout the Feed class
- Ensure dates have TEXT affinity in SQLite, in case it matters
- Add a userPreAuth setting for when relying on the Web server to do authentication
- Changed Data::$l to Data::$lang; it's not used enough to justify the possibly confusing shortening
- Made database auto-update a general rather than per-driver setting
- Added settings for forthcoming feed fetching service
- Revamped design of Query class to be more consistent and predictable, and generally suck less
- Removed special case for Query class in Statement class
- Cleaned up database schema somewhat
- Test the values returned by articleList
- Test paged and reversed listing
- Test marking, un/read, un/starred, and both
- Nearly complete rewrite of marking queries; previous efforts simply did not work, or overzealously updated or inserted
- Testing of marking contexts still required
Fixed many errors in queries:
- Prepared statement wasn't running at all :)
- Returning feed ID rather than subscription ID
- latestEdition wasn't matching
- Fixes #55
- Included test for Context
- Adjusted Database::editionLatest() to use Context
- Adjusted NCN handler and tests accordingly
- Also refined experimental Database::articleList() method and added experimental Database::articlePropertiesSet() method
- Driver->prepare() can now take a new Database\Query object as the first parameter
- The Query class allows for building a complex query out of a main body, common table expressions, WHERE coniditions, ordering conditions, a limit, and offset
- The subscriptionList method is the prototypical example of usage
- The experimental articleList method will in time be adapted to use this as well
- A generic means of specifying a selection context is forthcoming; this would remove subscriptionList's $folder and $id parameters, for instance
- Adjusted NCN handler to use the topmost folder for folderId; suppressed the actual folder so as not to pollute output with non-standard data
- Updated tests
The previous complexity is really not required. Also modified how settingSet() works to avoid a REPLACE query, which is not compatible with PostgreSQL.
- Two items with differing IDs but identical hashes were matching in the deduplicator; they shouldn't
- They would also match in the database matcher, and shouldn't
- The second-pass database matcher was overaggressively finding items as new
Three bugs:
- The parser wrapping was generating hashes for fallback values in absence of actual values for both URL and title (which is valid if obscure RSS), URL and content, or title and content; now fallback values are detected and empty strings used instead of hashes
- The deduplicator was assuming all three hashes would always exist, which is no longer the case
- The database matcher was making the same assumption as the deduplicator
- Makes use of PHP's internal Web server to deliver expected responses from a real server
- Windows batch file can be used to run tests (Linux and Mac test runners to come later)
- Added PHPUnit to dev dependencies
- Fixes #8
- Fixes #9
- Fixes #10
- Fixes #11
- Fixes #12
- Differentiated between a missing ID which is the subject of an action (e.g. the subscription when moving a subscription) and other missing IDs (e.g. the folder when moving a subscription)
- Enforced same rules for subscription titles as for folder names, save that null is valid for subscriptions
- URLs are now matched centrally against a whitelist
- %-encoded URLs are still handled correctly
- Dispatched methods now only handle one specific task
- Filler methods (e.g. versionPOST) are no longer required
- Unhandled URLs now return 501 Not Implemented rather than 404 Not Found; this removes some ambiguity in the semantics of 404
- Implemented all but one feed-related function (it's more ofan item function)
- Fixed time conversion for input into SQL; dates in PM were previously wrong
- Added miscellaneous tentative functions to Database to help with peculiarities of NCNv1; these may change
- Tests to come soon
- functions not related to user management now have the existence of the affected user checked in the authorizer, when the affected user differs from the actor
- User::authorizationEnabled() now nests: disabling twice and then enabling once leaves the authorizer disabled
- Disabling of the authorizer is now tested
- User tests now use a partial mock instead of relying on User::authorizationEnabled()
- Added authorizer tests against a missing user
- Removed folder tests related to missing users
- Also added more subscription tests
- Introduced "strict" binding types for use when inserting into NOT NULL columns: any null value supplied is always cast to the supplied type rather than passing through
- Fixed feed updating further
- Filled out full complement of subscription manipulation functions
- made folderPropertiesSet possibly return false
The type parameters of Db\Driver::prepare() and the parameters of Db\Statement::run() can now be arrays, which will be iterated over recursively to bind scalar values to the SQL statement.
This simplifies the construction of arbitrary UPDATE statements (the WHERE clause no longer needs to be taken into account) and should make it clearer what is happening in these cases.
It should also simplify the creation of IN() clauses down the road if they become necessary.
• Changed Feed to where the favicon is gotten when parsing so the
feed’s site URL is used instead of the feed’s for favicons. Some feeds
don’t have the same domain.
• Fixed a bug in Database.
- Simplified folder removal; now properly relies on foreign keys for dependency resolution
- simplified *propertiesSet() methods by offloading input validation and query building to a generic function
- Implemented function to get the properties of a single folder (useful for internal use)
- Implemented a function to set the properties of a folder
• Added category updating in Database->updateFeeds()
• Made Database->updateFeeds() fail quietly when exceptions are
occurred, logging them in the database instead of failing.
• Changed the categories table name from arsse_tags to arsse_categories.
• Added version constant in bootstrap
• Made the user agent string output information in a format similar to
a web browser with software version along with OS information and
architecture
- RuntimeData has now been replaced by a single static Data class
- The Data class has a load() method which fills the same role as the constructor of RuntimeData
- The static Lang class is now an instantiable class and is a member of Data
- All tests have been adjusted and pass
- The Exception tests no longer require convoluted workarounds: a simple mock for Data::$l suffices; Lang tests also use a mock to prevent loops now instead of using a workaround
• Started implementing feed updating (Database->updateFeeds())
• Moved hashing to the Feed object, now done when parsing
• Moved adding of articles to the database to its own method
(Database->articleAdd())
• Implemented a simple wrapper for picoFeed which fixes the id problems
and keeps error handling within its own class
• Updated Database.php to use the new class
• Replaced mentions of ownCloud with NextCloud in the schema
• Added hashes to schema for identification and change detection;
removed NextCloud hash and fingerprint; removed enclosure and category
hashes
- Result->lastId() added as MySQL and PostgreSQL have equivalent functionality
- Adjusted tests accordingly
- Cleaned up Database class to make use of this and getAll()
- Fixes #45
- Renamed getSingle to getValue to avoid possible confusion
- Added test to ensure getValue() always returns the first datum of each row rather than going column-to-column
- Introduced abstract Statement class to hold common methods
- Common methods currently consist of a date formatter and type caster
- Moved binding tests to a trait for reuse with future drivers
- User-related database methods will now throw User\Exception upon errors
- Internal userAdd method can now generate random passwords
- Pursuant to above, dependency on password genrator has been added, and password-related methods now return strings instead of booleans
- User class methods now all explicitly follow different branches for internal/external/missing implementations
- various User class methods now perform auto-provisioning of the internal database when external implementations report success on users not in the database
- Tests have been adjusted to account for the above changes
- Lots is probably still broken
• Fixed a bug where the feed object’s variable was being reused as the
id for adding feeds to subscriptions
• Modified inline documentation
• Added a TODO for populating the articles table when adding a new feed
As the User class depends on the database and this has yet to be tested (though I'm fairly certain it works), the mock driver also acts as a mock of the required database functions, with both instances sharing a common storage structure. Later test series should separate the two.
Tweaks:
- get() method can now report loaded and well as wanted locale
- msg() passed without vars still gets formatted to catch malformed strings
- set('en', false) followed by set('en', true) will now immediately load
- Lang::$synched was not getting set to true properly
Tests:
- Added test for get()
- Added test for malformed strings (exception code was missing)
- Added test for missing strings
- Added test for strings taking variables not being passed any variables
